search

dirac-institute / kbmod-wf

MIT License
0 stars 0 forks source link

Find solution for logging and results when multiple users are running workflows #25

Open drewoldag opened 1 month ago

drewoldag commented 1 month ago

As it stands, all the logs go to .../<date>/<run_id> and all the output files end up in .../staging/ but when multiple people are running at the same time this won't work. People would end up clobbering each other's run logs, and the checkpointing system wouldn't know where to look for a given user's latest checkpoints - instead it would consider everyones checkpoints.

Additionally, now that there are multiple types of workflows, we need a way to distinguish which workflow produced a particular set of results along with some way to distinguish who is producing those results. This is also a problem for the checkpoint files. It would be bad for workflow A to use a checkpoint from workflow B just because the input files were named the same thing.

Probably need something like .../<person>/<run_id> or .../<workflow_id>/<person>/<run_id>

drewoldag commented 1 month ago

@wilsonbb had some thoughts about a more substantial system for manifest handling when multiple people are processing from the same staging data.

https://github.com/dirac-institute/kbmod-wf/pull/30#discussion_r1718967456