search

seqeralabs / nf-tower

Nextflow Tower system
https://tower.nf
Mozilla Public License 2.0
147 stars 53 forks source link

Resume but with new code #241

Open apeltzer opened 4 years ago

apeltzer commented 4 years ago

Hi!

@jfy133 and me were debugging an issue on AWS recently and had to start a pipeline run, fix, re-run etc pp.

The way I understood it in the UI, the current resume feature in tower only resumes an existing pipeline at the previously executed code state, e.g. using the commit hash at the time the pipeline was executed the last time. If a user for example updates a pipeline locally / on github and wants to resume with that particular code state, this is currently not possible from within tower.

However it would be great to be able to do so - locally such an approach works well in 99% of all cases and only steps that aren't already done / finished are rerun, making debugging much faster :-)

jfy133 commented 4 years ago

To clarify, what it appeared to happen to us was: that the first time we launched a run with a branch name specified as a revision, this is converted to a specific commit. This was displayed in the in the first instance command line correctly. But when relaunched, the branch name was converted to the commit hash. Relaunching for a third time again appears not to have got the latest commits (with the fixes from the first relaunch). This is presumably not the behaviour we want as we want to resume with a later commit every time.

pditommaso commented 4 years ago

Wait, but you specify any revision you want even with resume. IS that not enough?

Screenshot 2020-07-29 at 14 55 45
jfy133 commented 4 years ago

Yeah I guess, but if you're not 'paying attention' and it's default displayed is the commit at the time of the original version of the branch, you may launch with the original commit again and not the new one. At least I'm pretty sure that's what @apeltzer and I expeirenced?

pditommaso commented 4 years ago

I understand, not sure how it could be improved tho. I think being a resume it should default to last used commit id. What would you suggest?

apeltzer commented 4 years ago

At least to me that you can actually enter the revision number there isn't/wasn't clear from the UI - having it prefilled with the hash of the previous run is fine (and actually quite sane to do so), but maybe we could highlight differently that one could also specify something else there? Making it not greyed-out?

blajoie commented 3 years ago

This is still an open issue no?

If you launch with a branch name, fail, and then go to relaunch. The previous git hash will be set in the revision number field. If you instead change that to the branch name, that has the latest commit, it seems to run with the new code, but then ignores all of the cached results. Are others seeing the same? If you resume using the existing code / revision number, then the caching + resume functionality works.

Is the guidance to instead always be using commit-ids in this field?