search

mmcdermott / MEDS_transforms

A simple set of MEDS polars-based ETL and transformation functions
MIT License
20 stars 5 forks source link

Lock files should be pipeline ID specific in some way -- this will enable pipelines to flag when old run locks are present. #194

Open mmcdermott opened 2 months ago

mmcdermott commented 2 months ago

Tagging @Oufattole as this will help MEDS-Tab be more usable too.

mmcdermott commented 2 months ago

This will be a bit tricky as it depends on hydra nonsense. Basically we need all workers to be able to have the same "pipeline ID" as assigned via the hydra config, meaning when you run a job with multirun, we need a variable to interpolate once before the sweeper goes, then the sweeper to go and not reset that variable. This can be a little finnicky with hydra interpolation rules. But I think some resolvers work like that -- in particular maybe the hydra specific ones do, like now. This would also preclude someone from manually launching multiple workers, which I am fine with, as I can't imagine a situation where that would be better than using hydra launchers.