microbiomedata / nmdc-aggregator

Scripts that periodically aggregate data related to KEGG search
0 stars 0 forks source link

Aggregation Scripts

These scripts generate the KEGG/COG/Pfam aggregations that are used for search.

A container hosted on Spin runs the agg.sh script, which performs the aggregations periodically (once every 4 hours, by default).

[!NOTE] The container image is hosted here.

Development

Here's how you can set up a local development environment:

Unless otherwise specified, all commands below are designed to be run from the root directory of the repository.

[!NOTE] These instructions do not cover the process of setting up a local MongoDB server or getting access to the NERSC filesystem.

  1. Create and activate a Python virtual environment
    python -m venv ./.venv
    source ./.venv/bin/activate
  2. Install Python dependencies
    pip install -r requirements.txt
  3. Done

Testing

We use pytest as our test framework.

Here's how you can run the tests:

Unless otherwise specified, all commands below are designed to be run from the root directory of the repository.

  1. Run the tests
    pytest
  2. See the test results in the console

Deployment

Here's how you can build a new version of the container image and push it to the GitHub Container Registry:

  1. On GitHub, create a new Release.
    1. Create a tag.
    2. Click the "Generate release notes" button.
    3. Leave the Release title empty (so GitHub reuses the tag name as the Release title).
    4. Click the "Publish release" button.
  2. Wait 3-4 minutes for the container image to appear on the GitHub Container Registry.

    Taking a long time? Check the "Actions" tab on GitHub to see the status of the GitHub Actions workflow that builds the image.

Now that the container image is hosted there, you can configure a Spin workload to run it.

Configuration

Environment variables

Release Notes

https://github.com/microbiomedata/nmdc-aggregator/releases