Open mpm896 opened 1 month ago
Hey @mpm896 - there's no functionality for this out of the box but you can always run on a subset of data https://warpem.com/reference/warptools/subsets/
We typically use tmux to get terminals which live beyond an ssh connection on remote systems
Might not be too hard to add though... --only_do_unfinished or something - I don't know how much that metadata check knows
@dtegunov what do you think?
Thanks for pointing me to the subset docs! One of the reasons I was asking about this also is because I didn't catch which dataset the job got cancelled on, I suspect one of the datasets is only partially complete. For this purpose I can write up a small script to search for the incomplete tiltstack directory and include that in the subset, but I thought it could be useful to include as a feature
@alisterburt It's easy to add for some things (especially those that already support partial reprocessing in desktop Warp), but in the case of (Ar)etomo, the success of a run is hard to track, especially if users try both tools. None of the solutions I can think of right now (e.g. add a "success" dummy file in the tilt stack folder) are pretty.
Given the above re: Aretomo at least, a 'no-overwrite' option could be easier to implement and IMO is 90% as good as a true 'unfinished only' feature.
Is it possible to restart a job from where it left off? For example, I'm running
ts_etomo_patches
on ~100 tilt series, which is taking some time, and I had to quit the job one night. It finished about 50, but when I restart the job it finds previous metadata for all 50, however starts from fresh and performs alignment on all of 100.I've tried submitting as a nohup background job (i.e.
nohup WarpTools ts_etomo patches ... 2>&1 > logfile.log &
) but somehow it gets stuck and never really starts the jobs (I've been finding this issue with some IMOD processes when sent to the background). Since this doesn't seem to work for me, is it possible to pickup a WarpTools job where it left off? Thanks!Matt