Unify dose not work

[Sungryong-Oh]

Hi, I recently start use monorail pipeline again, but fail to make a unify output. even though pump pipeline seems like to be successfully finished.

I pulled the most recent version of monorail-pump v.1.3. I tested two unify version -v.1.1.2rc and v.1.1 but both fail to finish successfully. I ran it with singularity image.

Here is my error log:

[Mon Aug 19 23:44:20 2024] rule split_final_rejoined_exons: input: all.exon_counts.rejoined.tsv.gz, all.exon_counts.rejoined.tsv.gz.accession_header output: exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.G026.gz, exon_sums_per_study/08/SRP 433808/sra.exon_sums.SRP433808.G029.gz, exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.R109.gz , exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.F006.gz, exon_sums_per_study/08/SRP433808/sra .exon_sums.SRP433808.ERCC.gz, exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.SIRV.gz jobid: 19 threads: 10

    /bin/bash /recount-unify/rejoin/split_out_exon_sums_by_study.sh sra G026,G029,R109,F006,

ERCC,SIRV 1709834 /container-mounts/ref/exon_bitmasks.tsv /container-mounts/ref/exon_bitmask_coords.tsv all.exon_counts.rejoined.tsv.gz 10 rm -rf exons_split_by_study_temp exon_annotation_split_runs

Error in tempfile() using template /scr1/users/ohs5/tmp/parXXXXX.par: Parent directory (/scr1/users/ohs5 /tmp/) does not exist at /usr/bin/parallel line 3993. [Mon Aug 19 23:44:20 2024] Error in rule split_final_rejoined_exons: jobid: 19 output: exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.G026.gz, exon_sums_per_study/08/SRP 433808/sra.exon_sums.SRP433808.G029.gz, exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.R109.gz , exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.F006.gz, exon_sums_per_study/08/SRP433808/sra .exon_sums.SRP433808.ERCC.gz, exon_sums_per_study/08/SRP433808/sra.exon_sums.SRP433808.SIRV.gz

RuleException: CalledProcessError in line 333 of /recount-unify/Snakefile: Command ' set -euo pipefail;
/bin/bash /recount-unify/rejoin/split_out_exon_sums_by_study.sh sra G026,G029,R109,F006, ERCC,SIRV 1709834 /container-mounts/ref/exon_bitmasks.tsv /container-mounts/ref/exon_bitmask_coords.tsv all.exon_counts.rejoined.tsv.gz 10 rm -rf exons_split_by_study_temp exon_annotation_split_runs ' returned non-zero exit status 2. File "/recount-unify/Snakefile", line 333, in __rule_split_final_rejoined_exons File "/opt/conda/envs/recount-unify/lib/python3.9/concurrent/futures/thread.py", line 52, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message

Hopefully to get an idea from you to fix it. Thanks!

[ChristopherWilks]

Hi @Sungryong-Oh,

Could you post your full unifier command line please? i.e. the command with all the parameters you're using to run the unifier.

Thanks, Chris

[Sungryong-Oh]

Here is the full command line of Unify run:

/bin/bash /home/ohs5/monorail-external/singularity/run_recount_unify.sh /home/ohs5/monorail-external/singularity_image/recount-unify_1.1.2rc.sif(or recount-unify-1.1.1.sif) hg38 /home/ohs5/monorail-external/GENOME/ /home/ohs5/monorail-external/output/unify /home/ohs5/monorail-external/output/pump/output/ /home/ohs5/monorail-external/output/metadata.tsv 10 sra:208

[ChristopherWilks]

Thanks, that looks fine.

Going back to error reported in your original post:

/tmp/) does not exist at /usr/bin/parallel line 3993```

The monorail unifier makes heavy use of the GNU Parallel program which appears to be the source of this error.
It's been quite a while since I ran this under Singularity and I don't usually test updates under it these days since my work has mostly been focused on Docker use cases.

Does the `/scr1/users/ohs5` path look familiar to you and do you have permissions to write there?

[Sungryong-Oh]

Dear Chris, Thanks for confirm.

Actually, I recently tried again and Unify worked well when I gave temporary directory only as /tmp. I realized that this is a kind of issue with "GNU Parallel". For some reason, GNU Parallel only works when I set TMPDIR as /tmp, not any other location. I set up TMPDIR under my personal one because I'm using it on hpc, but setting TMPDIR to other location seems not working at all.

Anyway, I don't know how to set the TMPDIR of GNUParallel when I'm using Unify, but it currently works and I successfully get an output. Please let me know if you can give me to change the setting of Parallel.

Thanks for your help.

[ChristopherWilks]

thanks for the update, feel free to (re)open or open a new ticket if you continue to run into issues