search

niaid / image_portal_workflows

Workflows related to project previously referred to as "Hedwig"
BSD 3-Clause "New" or "Revised" License
5 stars 1 forks source link

Callback output can be written to an output file #335

Closed annshress closed 1 year ago

annshress commented 1 year ago

Currently, we are accessing the callback output from the slurm logs. To make things easier, we could write it to a file instead.

Furthermore, slurm log file names could be much more accessible rather than devs solely relying on timestamps.

annshress commented 1 year ago

Currently, SlurmCluster is passed into flow as a decorator as a class.

Two arguments are required to create the log file are log_directory and name of the job, which are directly written into a job shell script based on dask-jobqueue code. And presumably this is happening even before we can access prefect flow run's name (which can be accessed using prefect.context.get_run_context() inside a flow).

Therefore, we are letting this go for now, unless we find a way to inject flow run's name into the sbatch script.