Open jbanusco opened 1 year ago
This actually makes sense considering how fair
and ignore
work.
The fair directive ensures that outputs are emitted in the order they were received, therefore it waits for each output to be produced in order. Therefore, since the ignore error strategy simply ignores errors, if you use them together, any error will cause the pipeline to hang forever.
Basically, the fair directive needs to know how to skip outputs that will never be produced because they failed.
By the way, I am curious as to why the fair directive is helping you with cache issues. Why would the order in which outputs are emitted by a process affect the task executions of downstream processes? I suspect you may have written your pipeline in a way that causes non-deterministic behavior. Unless you are explicitly generating random inputs at some point, your pipeline should execute the exact same tasks given the same inputs.
Ah, I see! Thank you very much for the clarification, I thought by default the fair directive would skip these outputs.
Regarding the cache issues, you are right. The problem was that the behavior was non-deterministic although I don't fully understand why. I was creating an initial channel with the 'configuration' files for the processing of each subject. Then, each process was generating a new output file indicating if that processing step was successful or not. The input of each step/process in the pipeline was the configuration file + the file of the previous process. So, I started joining the channel with the configuration files with the output of each process based on the subject id, but I was still getting some issues.
Right now I just output the configuration file as an additional output of each process. In this way I don't have to join the output with the 'configuration' channel anymore. This seems to work and prevent the non-deterministic behavior. Hope I was clear!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bug report
I have a workflow that runs a data processing pipeline that I would like to apply to several subjects. I have problems with the cache since some subjects that are already processed are being re-processed every time I run the workflow. Therefore, I tried to use the fair directive. The fair derivative seems to solve this issue, but I noticed a possible bug.
Expected behavior and actual behavior
In an HPC cluster [with slurm and local executor]: If I have 10 subjects and subjects 4-6 fail, the pipeline keeps going downstream for subjects 1-3, but not for subjects 7-10. It seems like all the processes that take place after an error are stopped. Given that I use the 'ignore' as error strategy I expected that all successful process will keep going downstream.
Locally in my workstation: I noticed that the behavior is different than in the HPC cluster. Here all the downstream processes are stopped.
Steps to reproduce the problem
You can find the code to reproduce the error in this repository: https://github.com/jbanusco/Fair_DummyTest_Bug Run the ./create_files.sh to generate some dummy files and then run the workflow.nf
Program output
HPC:
Local:
Environment