Luigi pipeline integration into this project - Githubissues

NERC-CEH / plankton_ml

A project for image processing and analysis pipelines for plankton sampling

GNU General Public License v3.0

0 stars 1 forks source link

Luigi pipeline integration into this project #38

Closed metazool closed 1 month ago

metazool commented 1 month ago

https://github.com/NERC-CEH/plankton_pipeline_luigi/ - temp project that needs integrated into this one
add a batch mode as there's an upload limit of 1K files
run the tests in CI, flush out issues with them (looks related to Luigi tasks not re-running when pytest creates a new tmp_path fixture... )

Benefits to keeping this all in the same project, even if it's a wee bit of extra work:

reproducible environment in one place (we could add pipeline-specific dependencies to pyproject.toml)
import classes and utils from cyto_ml to avoid keeping multiple copies of the code
Use the continuous integration pipelines in this repository (lint, run tests etc)
Ease of managing the repo (don't need to recreate permissions, policies)