nudup instances loop endlessly when run in parallel inside a container

[AdrianS85]

Dear all,

I was trying to run 20 bam files through nudup.py in parallel inside Singularity container. I used parallel processing as implemented in snakemake, nextflow and parallel GNU. The effect was always the same: some of the files were processed, while others ran (or just stay hanged?) forever. I have logged the paralell's GNU stdout/err during execution of nudup script - https://github.com/AdrianS85/varia/blob/master/nudup_raport.txt. Please notice that the same jobs are issued over and over again. For comparison, the strip script, which is run before nudup.py, runs in parallel properly - https://github.com/AdrianS85/varia/blob/master/strip_raport.txt

I will be thankful for any help, Adrian

EDIT: Program also hangs when running it in parallel outside the container. Ive also produced another raport for failed jobs: https://github.com/AdrianS85/varia/blob/master/nudup_errors_more

https://github.com/nugentechnologies/nudup/blob/468c62e948b87fa8a3231355d26cc37e1d0717d5/nudup.py#L1

[mlovci]

Hi @AdrianS85 - I am not familiar with Singularity or snakemake, but we can try to help. A few things to check:

• Based on the [samtools] <defunct> - please verify the version of samtools you're using matches the recommended versions on our main page. (note the --old-samtools flag is required for old samtools versions)
• Validate your temporary directory is writable for named pipes (it cannot be a mapped network drive)
• Validate your temporary directory has adequate space for writing and that it is not filling while you run in parallel (you can specify the temp directory with the -T option)
• Validate your implementation of GNU parallel is appropriate - try another command like echo to determine whether the loop exits as expected. (Can you provide the parallel command you issued here for inspection?)

[AdrianS85]

Dear mlovci,

• I use samtools 1.9, so I think that should not be an issue
• Temporary directory is writable, although when starting the parallel run, I sometimes get error: nudup.py: error: Must be able to make named pipes in temp directory. Seeman mkfifofor named pipe info.. Exactly the same command issued moment later starts without problem.
• Directory to which I point the temporary data has 6 tb of free space, I think more than enough for 20 ~2.5GB .bam files?
• Please have a look at my script here: https://github.com/AdrianS85/RRBS/blob/master/RRBS_follow_up_dedup.sh

I also tried to run only 5 files in parallel in case of RAM issues, but it still seems to work/fail randomly, sometimes after producing 16 files, sometimes after single one.

Best, Adrian