TimothyOlsson
commented
5 years ago Further test shows that the problem could be because of which Singularity version is used. When running the PID checker in singularity versions <v3.2, aka ps ux | grep PID in the shell, you get all process information for the user, but for v3.4, when you run the exact same command, you don't get the process information of the host.
This will require either one of two solutions:
- Only install singularity <version 3.2 with the shell script (this is a bad idea, since some clusters will update singularity, resulting in the bug persisting)
- Rework how running process tab works. This change in singularity might make it unpossible to kill processes in the host computer, but it is better than a broken system.
Edit: Running ps aux in version v3.4 yields no processes in the host computer
In some cases, the pipeline shows "Failed" even though the process is still running. It seems that the "running jobs" tab sometimes cannot find the PID process (even though it could easily be checked with
ps aux | grep PID), which makes the GUI set the job as "Failed". It could either be some kind of sync problem, where stdout.txt has not been created yet and the GUI not finding the PID process, at the same time the "running job" tab updates. This is a minor problem, since it does not impact the process itself, but should be investigated since it prevents the user from killing the job via the GUI.