Implement searching/filtering CatalogNext releases for a new content updater

[birchamp]

What is needed to find the latest releases available for a specific resource type?

Phases:

1. using CN and just looking at Door43-Catalog results

   • once this is implemented can do

2. able to choose organizations

   • able to choose organizations to download
   • getting latest resources will include all organizations
   • work on tool GL selection and show org

Needed:

Search:

• change to use catalog next API to get all the latest resources from all orgs
  • update sourceContentUpdater.updateCatalog()
  • check latest results by org

Downloading:

• add support to parse twl's - Phase 2?
  • in phase 1 we create the translationHelps tW from original language and merging the tW articles. Talk to Birch.
• update.sourceContentUpdater.downloadResources() with org support

Notes: Explore pivoting on resource type. <= just need to sort by subject type API: https://git.door43.org/api/catalog/swagger#/v5/v5Search

• looks like this will get all the resources: https://git.door43.org/api/catalog/v5/search?subject=Bible%2CAligned%20Bible%2CGreek_New_Testament%2CHebrew_Old_Testament%2CTranslation%20Words%2CTranslation%20Notes%2CTranslation%20Academy&sort=subject&page=1&limit=50
• looks like we have both TSV Translation Notes and Translation Notes being returned
• currently the matches are using Like comparison, so lots of OBS items being returned also. @richmahn is looking into adding an exact match option.

Test Assumptions:

• the CN results for Door43-Catalog are the same as pivoted.json?

[PhotoNomad0]

@mannycolon Here are some tCore storage options for handling orgs:

current example ~/translationCore/resources/en/translationHelps/translationNotes/v33

option 1 - add org to version: translationNotes ↳ v33_unfoldingWord ↳ v34_test-org

option 2 - add org to resource ID: translationNotes_unfoldingWord ↳ v33 translationNotes_test-org ↳ v34

option 3 - add org as new layer: unfoldingWord ↳ translationNotes ↳ v33 test-org ↳ translationNotes ↳ v34~~

moved to https://github.com/unfoldingword/translationcore/issues/7187

[mannycolon]

I prefer option 1, we do a lot of filesystem calls in our codebase to check for resources and would prefer not to add an additional directory to overcomplicate this process. What do you think?

[mannycolon]

@PhotoNomad0

[richmahn]

Exact match option has been added. (actually it is the default, if you want a partial match, then you use partialMatch=1)

[PhotoNomad0]

Notes for development:

Need to add to tc-source-content-updater:

• support for multiple repo owners:
  • save owner in file system - use encodeURIComponent() to sanitize?
  • get latest version to have optional parameter for repo owner?
• updated Languages list returned by getLatestResources needs updating:
  • to include latest resources under each language.
  • to support multiple owners.
• latest remote catalog list needs to include old dcs catalogs as well as catalog next since not everything in the old catalog has been released, but priority given to catalog next.
  • support for twls <- made issue for this.
• original language resources are D43 Catalog only ...

[PhotoNomad0]

Need to do two searches since not all resources in D43-Catalog have been released on DCS.

1. We use the v5 API to get the released resources (stage==prod).
2. There is a v3 endpoint that gives us the entries from the old catalog.json

Then we merge the two lists removing entries in # 2 that are duplicates

[PhotoNomad0]

Done, back end work not QA testable