pVACsec fails on calculating manufacturability for MHC class II

[mantczakaus]

Hi, I am running the entire nextNEOpi pipeline on the TESLA dataset https://doi.org/10.1016/j.cell.2020.09.015 Currently, I'm experiencing issues with pVACsec for data from patient1, specifically some MHC class II alleles. I run it before on WES data of clear cell renal cell carcinoma tumor and matched normal (https://www.ebi.ac.uk/ena/browser/view/SAMEA4074323) and I did not encounter any issues.

For the patient I, MHC class I alleles worked fine. Some MHC class II were skipped (e.g. Allele DQA1*05:05 not valid for Method NetMHCIIpan. Skipping.) but for the ones that weren't skipped, pVACsec ended in error (below). I tested separately in a container, on 150GB disk.

  File "/opt/conda/bin/pvacseq", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.8/site-packages/pvactools/tools/pvacseq/main.py", line 116, in main
    args[0].func.main(args[1])
  File "/opt/conda/lib/python3.8/site-packages/pvactools/tools/pvacseq/run.py", line 158, in main
    pipeline.execute()
  File "/opt/conda/lib/python3.8/site-packages/pvactools/lib/pipeline.py", line 506, in execute
    PostProcessor(**post_processing_params).execute()
  File "/opt/conda/lib/python3.8/site-packages/pvactools/lib/post_processor.py", line 32, in execute
    self.calculate_manufacturability()
  File "/opt/conda/lib/python3.8/site-packages/pvactools/lib/post_processor.py", line 55, in calculate_manufacturability
    CalculateManufacturability(self.input_file, self.manufacturability_fh.name, self.file_type).execute()
  File "/opt/conda/lib/python3.8/site-packages/pvactools/lib/calculate_manufacturability.py", line 63, in execute
    writer.writerow(line)
  File "/opt/conda/lib/python3.8/csv.py", line 154, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
OSError: [Errno 28] No space left on device

Logs from the nextNEOpi pipeline attached command.err.txt command.log.txt command.out.txt command.run.txt command.sh.txt

[riederd]

Hi, the error message states, that you seem to be running out of space. You might need to check which of your fs mounts is affected. It might also be a TMP dir, e.g. SINGULARITYENV_TMPDIR

[mantczakaus]

Hi, thank you! Turns out I didn't have any of the environmental variables set from this line: set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_NXF_DEBUG=${NXF_DEBUG:=0}

I'm doing this now - fingers crossed! :)

[mantczakaus]

Hi @riederd I don't think I set everything that needs to be set, I'm still getting the"No space left on device error" when I run the pipeline but I did manage to run pvacsec from the container successfuly if I run it without the line "set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_NXF_DEBUG=${NXF_DEBUG:=0}"in front of the singularity shell command. I was wondering if you are able to see what I'm missing? Here is what I did: 1) I put the following lines in the .bashrc file and I made sure all these folders existed

export NXF_TEMP=/scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP
export SINGULARITYENV_TMP=/scratch/project_mnt/S0091/mantczak/.singularity/SINGULARITYENV_TMP
export SINGULARITYENV_TMPDIR=/scratch/project_mnt/S0091/mantczak/.singularity/SINGULARITYENV_TMPDIR
export SINGULARITYENV_NXF_DEBUG=/scratch/project_mnt/S0091/mantczak/.singularity/SINGULARITYENV_NXF_DEBUG   

2) I run the following command to get inside of the image set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_NXF_DEBUG=${NXF_DEBUG:=0} singularity shell -B /scratch/project_mnt/S0091/mantczak -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/iedb --no-home --containall -H /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP -B /scratch/project_mnt/S0091/mantczak/pipelines/nextNEOpi/assets -B /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources -B /scratch/project_mnt/S0091/mantczak/soft/hlahd.1.7.0 -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/iedb:/opt/iedb -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/mhcflurry_data:/opt/mhcflurry_data /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_SINGULARITY_CACHEDIR/apps-01.i-med.ac.at-images-singularity-pVACtools_3.0.2_icbi_3d50ac02.sif 3) I entered the folder work folder where the command was executed, e.g. cd /scratch/project_mnt/S0091/mantczak/tests/nextneopi_validation/work/57/20ba4c0b3bf5b64555cc13c51c7d7a (I also removed the MHC_Class_II folder from this work directory) 4) I executed the command: bash .command.sh Result: This finished with the "No space left on device error" error. Now, if in step 2) you run singularity shell -B /scratch/project_mnt/S0091/mantczak -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/iedb --no-home --containall -H /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP -B /scratch/project_mnt/S0091/mantczak/pipelines/nextNEOpi/assets -B /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources -B /scratch/project_mnt/S0091/mantczak/soft/hlahd.1.7.0 -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/iedb:/opt/iedb -B /QRISdata/Q5952/data/nextNEOpi_1.3_resources/databases/mhcflurry_data:/opt/mhcflurry_data /scratch/project_mnt/S0091/mantczak/.nextflow/NXF_SINGULARITY_CACHEDIR/apps-01.i-med.ac.at-images-singularity-pVACtools_3.0.2_icbi_3d50ac02.sif so the command without set +u etc. prefix then pvacsec runs without problems.

Calculating Manufacturability Metrics
Completed
Running Binding Filters
Completed
Running Coverage Filters
Completed
Running Transcript Support Level Filter
Complete
Running Top Score Filter
Completed

Done: Pipeline finished successfully. File /scratch/project_mnt/S0091/mantczak/tests/nextneopi_validation/work/57/20ba4c0b3bf5b64555cc13c51c7d7a/MHC_Class_II/patient1_tumor.filtered.tsv contains list of filtered putative neoantigens.

So I'm wondering if you see something I don't see - should I set some other environmental variables?

[riederd]

This is strange. Can you see any differences in the output of env, mount and df -h when entering the container as in 2. compared to the way you did in 4. ?

[mantczakaus]

Hi @riederd thanks for the reply! I run everything you asked for, I'm attaching all the files. Suffix _set means the set +u... command was added like in point 2), suffix _noset means it was run like in point 4). df_noset.txt df_set.txt env_noset.txt env_set.txt mount_noset.txt mount_set.txt After examining env_... files I thought I should have also set $TMP and $TMPDIR. I've noticed that if $TMPDIR is not set, it gets automatically assigned /tmp upon entering nodes on the HPC I use. I have added them to the .bashrc

export NXF_TEMP=/scratch/project_mnt/S0091/mantczak/.nextflow/NXF_TEMP
export SINGULARITYENV_TMP=/scratch/project_mnt/S0091/mantczak/.singularity/SINGULARITYENV_TMP
export SINGULARITYENV_TMPDIR=/scratch/project_mnt/S0091/mantczak/.singularity/SINGULARITYENV_TMPDIR
export TMPDIR=/scratch/project_mnt/S0091/mantczak/.tmp
export TMP=/scratch/project_mnt/S0091/mantczak/.tmp

But that hasn't improved things. When I re-run the pipeline it failed on the same steps with the same "no space" error. Do you have any ideas?

[riederd]

Hmm, you are running the pipeline on a cluster, so I'd assume that the particular compute node on which it gets executed has not enough space available in /tmp.

Can you try to add the following lines to nextNEOpi.nf just before https://github.com/icbi-lab/nextNEOpi/blob/aac260dbd5da701d22a09846045f3f42981cd4a1/nextNEOpi.nf#L5546

mkdir -p ${tmpDir}
export TMPDIR=${tmpDir}

and then either run the pipeline with --tmpDir /scratch/project_mnt/S0091/mantczak/.tmp or set tmpDir in conf/params.config

pVACseq uses the python tempfile module to set where temporary output gets written to. So it should honor the $TMPDIR env variable (if the directory exists). See https://docs.python.org/3.10/library/tempfile.html#tempfile.gettempdir

I have no idea why it is not taken from the existing env, I'd expect it would....

[mantczakaus]

Thank you @riederd - this worked! I had four alleles for which pVAC failed and now it's two but it fails for a different reason so I consider this problem to be solved!

[riederd]

Great!