Process `snp:aggregate_all_variants (1)` terminated -> unsetting TMPDIR fixed it

[sklages]

When trying to run the workflow with the demo dataset I get (very late) an error, which has also been mentioned in #108 .. but was not further discussed.

ERROR ~ Error executing process > 'snp:aggregate_all_variants (1)'

Caused by:
  Process `snp:aggregate_all_variants (1)` terminated with an error exit status (2)

Command executed:

  prefix="merge"
  phase_vcf=false
  if [[ $phase_vcf == "true" ]]; then
      prefix="phased"
  fi
  ls merge_output/*.vcf.gz | parallel --jobs 4 "bgzip -d {}"

  pypy $(which clair3.py) SortVcf             --input_dir merge_output             --vcf_fn_prefix $prefix             --output_fn MY_SAMPLE.wf_snp.vcf             --sampleName MY_SAMPLE             --ref_fn demo.fasta             --cmd_fn CMD             --contigs_fn CONTIGS

  if [ "$( bgzip -fdc MY_SAMPLE.wf_snp.vcf.gz | grep -v '#' | wc -l )" -eq 0 ]; then
      echo "[INFO] Exit in all contigs variant merging"
      exit 0
  fi

  # TODO: this could be a separate process
  if [ "false" == "true" ]; then
      pypy $(which clair3.py) SortVcf                 --input_dir merge_outputs_gvcf                 --vcf_fn_prefix merge                 --vcf_fn_suffix .gvcf                 --output_fn tmp.gvcf                 --sampleName MY_SAMPLE                 --ref_fn demo.fasta                 --cmd_fn CMD                 --contigs_fn CONTIGS

          # Reheading samples named "SAMPLE" to params.sample_name. If no
          echo "SAMPLE" "MY_SAMPLE" > rename.txt
          bcftools reheader -s rename.txt tmp.gvcf.gz > MY_SAMPLE.wf_snp.gvcf.gz
          bcftools index -t MY_SAMPLE.wf_snp.gvcf.gz && rm tmp.gvcf.gz rename.txt
  fi

  echo "[INFO] Finish calling, output file: merge_output.vcf.gz"

Command exit status:
  2

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    gocryptfs not found, will not be able to use gocryptfs
  Error in tempfile() using template /scratch/local2/USER.tmp/parXXXXX.par: Parent directory (/scratch/local2/USER.tmp/) does not exist at /home/epi2melabs/conda/bin/parallel line 5150.

Work dir:
  /scratch/local2/USER/nextflow/wf-human-variation/output_nogpu/workspace/39/ce770c5b99847f3307e1689757856f

When I unset TMPDIR (in my environment, where I start the workflow) the workflow finishes successfully.

Any idea on why the error occurs or why $TMPDIR is an issue here?

[SamStudio8]

Hi @sklages, it would appear that the TMPDIR from your execution shell is forwarded by Nextflow through to the container environment, but the container will not have permission to access it (as it has not been bind mounted). Unsetting TMPDIR will cause the container to use its own temporary directory.

[trum994]

I'm getting this exact "Error in tempfile() using template" error. So I tried the recommendation here of unsetting TMPDIR. However this made no difference in my situation. I also went into to the .command.sh and added two echo lines:

echo "tmpdir: $TMPDIR"
echo "singularity tmp dir: $SINGULARITY_TMPDIR"

which resulted in:

tmpdir: /local/26572187
...: line 3: SINGULARITY_TMPDIR: unbound variable

I'm using this config

singularity {
    enabled = true
    autoMounts = true
    runOptions = '-B $SINGULARITY_TMPDIR:/tmp'
    //Used to allow Singularity to access bashrc variables
    envWhitelist = 'SINGULARITY_TMPDIR'
}

How can I debug this? Any help is appreciated.

[Y-Pei]

Hi,

I'm getting the exact same "Error in tempfile() using template" error and unsetting the TMPDIR did not help as the TMPDIR didn't seem to be set in the first place.

Could you please advice on how to proceed?

Manhy thanks!