Make API v2 Build from API v3

[jag3773]

Story

As a developer|admin I want to have API version 2 catalog files be generated from API version 3 data so that we don't have extra dependencies/time spent on version 2 and so that we ensure that we are backwards compatible.

Notes

This will force us to get all our ducks in a row in terms of content in the v3 API as well. This needs to include automated acceptance testing too.

• [x] First acceptance test is comparing the JSON generated by this convertor against the existing v2 JSON file(s).
• [x] Second acceptance test is comparing JSON generated by this convertor against the structure of the v2 API. Note that this test will continue to be useful in the future whereas the first one is only useful until we publish new content. This should include:
  • ~[ ] Tests that no new keys are added~ This is not necessary - @neutrinog
  • [x] Tests that no existing keys are removed
  • [x] Tests that file structure remains the same (multiple directories in the /ts/v2/ namespace)

[da1nerd]

Updating the v3 api to follow the rc 0.2 spec and api structure defined at https://github.com/unfoldingWord-dev/door43.org/wiki/API-v3-Resource-Catalog-Endpoint. I also pushed the RC version of en-obs into Door43-Catalog so we can begin writing the API 2 port once the updates above are finished.

[da1nerd]

just forked the following repos into Door43-Catalog. Need to review the catalog building to make sure things are added correctly.

• https://git.door43.org/Door43-Catalog/en-obs-tq
• https://git.door43.org/Door43-Catalog/en-tw
• https://git.door43.org/Door43-Catalog/en-tq
• https://git.door43.org/Door43-Catalog/en-ta

[da1nerd]

@jag3773 no webhooks are fired when forking a repo into Door43-Catalog. This means forked repos will not be available in the catalog until a commit is added after it has been forked.

[richmahn]

Yes, I verified that a repo create event in an org doesn't trigger the organization webhook. This does work on github, so not sure why this wasn't implemented in Gogs/Gitea....I looked for an issue for it and couldn't find anything.

[richmahn]

Two solutions: 1) Get Ethan (or any of us) to add the Organization webhook for repo creation/deletion and/or 2) have a cron job run every 5 minutes to see if a Door43-Catalog repo is not in the catalog and trigger a commit webhook if not.

[da1nerd]

A few things I'm setting up today:

• [x] create cron job
• [x] convert en-udb to RC
• [x] convert en-ulb to RC
• [x] fix versification id in merged chunks file.
• [x] add tests for ulb/udb resources once they are in RC format.
• [x] fix catalog processing versification
• [x] update signing function to simply approve the versification
• [x] create new lamda function to check for new forks

[da1nerd]

@richmahn @jag3773 I've run into a road block for manually finding new repos via the cron job. This api end point is supposed to return the public repositories of the organization https://git.door43.org/api/v1/users/Door43-Catalog/repos but I get an error instead.

I created a cron job and stubbed out a new function to check for new repos but without the api working I can't go further.

[jag3773]

Forks do show up on the /explore page... can we hit that endpoint and then filter for anything in Door43-Catalog? I know it's kludgy, but we don't have much choice in the near term.

[da1nerd]

super kludgy, and it would take a lot of work to parse the html and handle the pager. I still have a lot to do with the api so I'll punt on this for now.

[jag3773]

I meant hit the corresponding api endpoint for /explore...

[da1nerd]

that api end point above seems to be working now.

[da1nerd]

In the conversion process we need to use the old chunk markers. See #259

Update

This has been addressed.

[da1nerd]

This is done except I need to debug why the tS api generation is now getting stuck.

[da1nerd]

This is not exactly a blocker but it would be super nice to have this PR merged in https://github.com/unfoldingWord-dev/USFM-Tools/pull/38. USFM tools produces so much log output that it's VERY difficult to find anything meaningful. This PR would allow controlling log output.