search

warpem / warp

https://warpem.github.io/warp/
GNU General Public License v3.0
35 stars 7 forks source link

request for ability to continue unfinished processing in WarpTools #187

Open mpm896 opened 1 month ago

mpm896 commented 1 month ago

Is it possible to restart a job from where it left off? For example, I'm running ts_etomo_patches on ~100 tilt series, which is taking some time, and I had to quit the job one night. It finished about 50, but when I restart the job it finds previous metadata for all 50, however starts from fresh and performs alignment on all of 100.

I've tried submitting as a nohup background job (i.e. nohup WarpTools ts_etomo patches ... 2>&1 > logfile.log &) but somehow it gets stuck and never really starts the jobs (I've been finding this issue with some IMOD processes when sent to the background). Since this doesn't seem to work for me, is it possible to pickup a WarpTools job where it left off? Thanks!

Matt

alisterburt commented 1 month ago

Hey @mpm896 - there's no functionality for this out of the box but you can always run on a subset of data https://warpem.com/reference/warptools/subsets/

We typically use tmux to get terminals which live beyond an ssh connection on remote systems

alisterburt commented 1 month ago

Might not be too hard to add though... --only_do_unfinished or something - I don't know how much that metadata check knows

@dtegunov what do you think?

mpm896 commented 1 month ago

Thanks for pointing me to the subset docs! One of the reasons I was asking about this also is because I didn't catch which dataset the job got cancelled on, I suspect one of the datasets is only partially complete. For this purpose I can write up a small script to search for the incomplete tiltstack directory and include that in the subset, but I thought it could be useful to include as a feature

dtegunov commented 1 month ago

@alisterburt It's easy to add for some things (especially those that already support partial reprocessing in desktop Warp), but in the case of (Ar)etomo, the success of a run is hard to track, especially if users try both tools. None of the solutions I can think of right now (e.g. add a "success" dummy file in the tilt stack folder) are pretty.

asarnow commented 1 month ago

Given the above re: Aretomo at least, a 'no-overwrite' option could be easier to implement and IMO is 90% as good as a true 'unfinished only' feature.