These are the AWS Lambda functions for generating the API catalog endpoint from the Door43 Catalog organization in our Door43 Git Service.
pip install -r requirements.txt
apex --env prod deploy
to publish everything to productionapex --env prod deploy ts_v2_catalog
for example to publish a single function.When a new repository is added or forked into the Door43 Catalog organization a chain reaction is started that eventually adds the content into the API, assuming all the checks passed. Here is an overview:
webhook
function which queues the latest git commit for processing.The next few functions run on a fixed schedule. If errors occur they are reported and the process resumed at the next scheduled run.
If a function produces errors 4 times in a row an email is sent to administrators.
signing
function looks for and signs new things in the queue.catalog
function takes everything in the queue and generates a new api catalog file. The content is now in the API!ts_v2_catalog
function converts the API catalog file into the legacy translationStudio API.uw_v2_catalog
function converts the API catalog file into the legacy unfoldingWord App Catalog.fork
function checks to see if new repositories exist in the organization and executes the webhook
function if necessary.The content in step (1) is now available in all three API endpoints.
acceptance
function runs when the catalog file is saved in step (4) above. And performs acceptance tests on the file to ensure it was generated correctly.The following provides a functional description of the functions in this repository.
Runs when a change is made in the Door43 Catalog
This function is run on a schedule and does the following:
This function is run on a schedule and does the following:
After a new catalog file is written to S3, this function does the following:
Technically this is all duplicate testing of what we are already doing elsewhere in the pipeline. This function is the "oops" catcher.
This function is run on a schedule and does the following:
dirty
.This function is run on a schedule and does the following:
This function is run on a schedule and does the following:
This function is run via AWS cron every 5 minutes and does the following:
Here's a high level overview of the AWS configuration. For Swagger definitions look in the aws_configuration folder. You can create an API in API Gateway by importing these Swagger definitions.
/webhook
/lambda/catalog
/lambda/fork
/lambda/signing
/lambda/ts-v2-catalog
/lambda/uw-v2-catalog
For example you can trigger the fork lambda at https://api.door43.org/v3/lambda/fork
.
The functions are not designed to always return useful information in the browser and may timeout, however they are still running properly.
The name of the stage in API Gateway determines the operating environment.
If the stage name begins with prod
the functions will operate on production databases.
If the stage name begins with anything other than prod
the functions will
prefix databases with the stage name.
For example:
prod
would use the d43-catalog-errors
db for reporting errors.dev
would use the dev-d43-catalog-errors
db for reporting errors.test
would use the test-d43-catalog-errors
db for reporting errors.Stage variables are configured within the stage defined in API Gateway.
These variables are accessible within lambdas from the event
parameter.
e.g. event['stage-variables']
cdn_bucket
cdn_url
to_email
from_email
api_bucket
api_url
gogs_url
gogs_org
gogs_token
log_level
how noisy the logger should be. debug|info|warning|errorversion
the api versionThe acceptance
function is ran according to a CloudWatch rule which runs when the catalog file is added to the api S3 bucket.
The trigger
function is ran according to a CloudWatch rule which is configured to run every 5 minutes via a cron job.
The following database tables are used by the API pipeline described above. Please note additional tables may be necessary when catering to multiple stages (described above).
d43-catalog-errors
tracks errors encountered in functions. Keyed with lambda
.d43-catalog-in-progress
tracks items in the queue. Keyed with repo_name
.d43-catalog-running
tracks functions that are running. This prevents certain functions from having multiple instances running at the same time. Keyed with lambda
.d43-catalog-status
tracks the status of the catalog generation. Keyed with api_version
.This tool will convert a csv file containing Greek words to USFM 3 format. You may execute the following command to learn how to use the tool.
python execute.py csvtousfm3 -h
This tool will inject tW links into the USFM generated by csvtousfm3
.
This tool is designed to replace the functionality of the config.yaml found within a tW RC
with the newly generated USFM3 content.
As such this is mostly a one time use tool.
If you are not sure what to use this tool for you probably shouldn't use it.
You may execute the following command to learn how to use the tool.
python execute.py maptwtousfm3 -h
This tool will convert a directory of OSIS files (xml) to a new directory of USFM3 files.
You may execute the following command to learn how to use the tool.
python execute.py osistousfm3 -h
You can run tests be executing the following:
python -m unittest discover -s tests
In order to deploy to production you need to run this command.
apex deploy --env prod
You can also deploy a specific function with
apex deploy --env prod catalog
If you want to cause a catalog to re-build you can delete the catalog entry from the d43-catalog-status
.
It will begin re-building within 5 minutes. Or you can try to force a re-try now by visiting https://api.door43.org/v3/lambda/catalog.
The lambdas are not allowed to run too often, so if you are trying to re-start the catalog lambda right away
you may also need to delete the d43-catalog_catalog
record from the d43-catalog-running
table.