The clean up steps should remove all of the temporary data from the cluster as a final step in the process -- fortunately, this temporary data is contained inside an identified working directory.
Clean up steps should also be run on error handling, so that partial temporary data does not cause clutter if the script fails for some reason.
The script should have a flag that prevents the clean up steps from running, which we will want during testing and debugging.
Consider transferring the slurm segtrain-####.out files to the VMs rather than deleting, so that these can serve debugging purposes as needed.
Note that in the API, Task Report Instances have a messages attribute (not read_only) -- consider sending the text of segtrain-####.out there. If you have a task report instance with a value for messages, you can see what that looks like by going to https://test-htr.lib.princeton.edu/quotas/[TASK_PK] (inputting the relevant TASK_PK).
Summarizing some considerations to keep in mind:
segtrain-####.outfiles to the VMs rather than deleting, so that these can serve debugging purposes as needed.Note that in the API, Task Report Instances have a
messagesattribute (not read_only) -- consider sending the text ofsegtrain-####.outthere. If you have a task report instance with a value formessages, you can see what that looks like by going tohttps://test-htr.lib.princeton.edu/quotas/[TASK_PK](inputting the relevant TASK_PK).