aind-codeocean-pipeline-monitor

Package for starting a pipeline, waiting for it to finish, and optionally capturing the results as a data asset.

Usage

• Define job using PipelineMonitorJobSettings class.
• Define a CodeOcean client.
• Construct a PipelineMonitorJob with these settings.
• Run the job with the run_job method.

from aind_codeocean_pipeline_monitor.job import PipelineMonitorJob
from aind_codeocean_pipeline_monitor.models import (
    CaptureSettings,
    PipelineMonitorSettings,
)

from codeocean.capsule import Capsules
from codeocean.data_asset import DataAssets
from codeocean.computation import (
    Computations,
    DataAssetsRunParam,
    RunParams,
)
from codeocean import CodeOcean
import os
from urllib3.util import Retry
from requests.adapters import HTTPAdapter

domain = os.getenv("CODEOCEAN_DOMAIN")
token = os.getenv("CODEOCEAN_TOKEN")
client = CodeOcean(domain=domain, token=token)
# Recommend adding retry strategy to requests session
retry = Retry(
    total=5,
    backoff_jitter=0.5,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504],
)
adapter = HTTPAdapter(max_retries=retry)
client.session.mount(domain, adapter)
client.capsules = Capsules(client.session)
client.computations = Computations(client.session)
client.data_assets = DataAssets(client.session)

# Please consult Code Ocean docs for info about RunParams and DataAssetParams
settings = PipelineMonitorSettings(
    run_params=RunParams(
        capsule_id="<your capsule id>",
        data_assets=[
            DataAssetsRunParam(
                id="<your input data asset id>",
                mount="<your input data mount>",
            )
        ],
    ),
    capture_settings=CaptureSettings(
        tags=["derived"]
    ),  # 'tags' is the only required field
)

job = PipelineMonitorJob(job_settings=settings, client=client)
job.run_job()

Installation

To use the software, in the root directory, run

pip install -e .

To develop the code, run

pip install -e .[dev]

Contributing

Linters and testing

There are several libraries used to run linters, check documentation, and run tests.

• Please test your changes using the coverage library, which will run the tests and log a coverage report:

coverage run -m unittest discover && coverage report

• Use interrogate to check that modules, methods, etc. have been documented thoroughly:

interrogate .

• Use flake8 to check that code is up to standards (no unused imports, etc.):

  flake8 .

• Use black to automatically format the code into PEP standards:

  black .

• Use isort to automatically sort import statements:

  isort .

Pull requests

For internal members, please create a branch. For external members, please fork the repository and open a pull request from the fork. We'll primarily use Angular style for commit messages. Roughly, they should follow the pattern:

<type>(<scope>): <short summary>

where scope (optional) describes the packages affected by the code changes and type (mandatory) is one of:

• build: Changes that affect build tools or external dependencies (example scopes: pyproject.toml, setup.py)
• ci: Changes to our CI configuration files and scripts (examples: .github/workflows/ci.yml)
• docs: Documentation only changes
• feat: A new feature
• fix: A bugfix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• test: Adding missing tests or correcting existing tests

Semantic Release

The table below, from semantic release, shows which commit message gets you which release type when semantic-release runs (using the default configuration):

Documentation

To generate the rst files source files for documentation, run

sphinx-apidoc -o docs/source/ src

Then to create the documentation HTML files, run

sphinx-build -b html docs/source/ docs/build/html

More info on sphinx installation can be found here.