mrinalwahal
commented
5 years ago Drone allows me to detach (see drone_docs) certain pipeline steps and run them in the background and immediately continue to the next step. So these steps essentially failing wouldn't fail the entire pipeline. The only problem I am considering is - these steps will proceed in the background, parallel with the next steps. And in essence - I want to run these steps in the same order.
For example - if I run the snakemake step under detached mode, my config would essentially run the next step i.e. back-push to the gin-proc branch at the same time. And it could be (and will be) easily possible that snakemake hasn't even produced its output at the time when the next step was back-pushing. Or anything could happen - like snakemake failing the workflow entirely.
Moreover, I really don't think there are any steps that we may essentially need to run detached. Because as of now, we are only allowing the users one pipeline step where their workflow is executed. Rest all the steps are designed by us in the required flow.
At a later stage, whence you allow the users multiple steps, then we'll need a work-around this.
achilleas-k
What happens with the build if a step in the drone.yml produces an error?
It should probably handle errors differently depending on the build step: setup, processing, output handling, etc. For example, if a user specifies some files to be pushed back on success but some of the files don't exist, it should probably warn that some of the specified files don't exist but it shouldn't fail completely (or at least, the files that do exist should be pushed).