search

icbi-lab / NeoFuse

NeoFuse is a user-friendly pipeline for the prediction of fusion neoantigens from tumor RNA-seq data.
GNU General Public License v3.0
17 stars 9 forks source link

YARA temp directory issue #16

Closed alhafidzhamdan closed 2 years ago

alhafidzhamdan commented 2 years ago

Hi there, I'm trying to run NeoFuse using singularity within an HPC.

INFO:    Convert SIF file to sandbox...
[-------------------------------- [NeoFuse] --------------------------------]

[12:53:31]  Paired End (PE) Reads detected: commencing processing
[12:53:31]  Processing files b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R1.fastq.gz - b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R2.fastq.gz
[12:53:31]  STAR Run started
[12:53:31]  Arriba Run started
[13:31:19]  YARA Run started
Couldn't create temporary file /local/22273612.1.igmm_long/SQNvM4n1B. (No such file or directory)
/usr/local/bin/seqan/include/seqan/file/string_mmap.h:635 FAILED!  (Memory Mapped String couldn't open temporary file)

stack trace:
  0    [0x555e04bb755e]  seqan::ClassTest::fail() + 0xe
  1    [0x555e04cbc824]  yara_mapper(+0x341824)
  2    [0x555e04d05e17]  bool seqan::_remap<unsigned long, seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long>, unsigned long>(seqan::String<unsigned long, seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > >&, unsigned long) + 0x47
  3    [0x555e04d07504]  SeqStore<void, YaraContigsConfig<seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > > >::SeqStore() + 0x124
  4    [0x555e04b4c4e9]  yara_mapper(+0x1d14e9)
  5    [0x555e04a90c15]  main + 0x5c5
  6    [0x2b6df93ebbf7]  __libc_start_main + 0xe7
  7    [0x555e04a90eca]  _start + 0x2a

[13:31:19]  OptiType Run started
An error occured during OptiType run, check /exports/igmm/eddie/Glioblastoma-WGS/RNA-seq/results/NeoFuse/C3N-03473T1_primary_tumour/LOGS/C3N-03473T1_primary_tumour.optitype.log for more details
INFO:    Cleaning up image...

I'm using the latest version 

./NeoFuse
NeoFuse v1.2.1

Here's my script:


module load singularity

$NEOFUSE_SCRIPT \
    -1 ${FASTQ_FILE}_R1.fastq.gz \
    -2 ${FASTQ_FILE}_R2.fastq.gz \
    -s $STAR_INDEX_DIR \
    -g $STAR_GENOME \
    -a $STAR_GTF \
    -o $OUTPUT_DIR \
    -d $RUN_ID \
    -V GRCh38 \
    -n 20 \
    -m 8 \
    -M 11 \
    -c L \
    --singularity

Any ideas? Thanks in advance!

A

abyssum commented 2 years ago

Hello @alhafidzhamdan,

This is probably due to YARA being unable to create a temp file because your system doesn't have a /temp/ directory. Have you tried the suggested workaround in #3?

Try to include $ export TEMPDIR=/path/to/tempDir/ in your batch file and tell me if this works for you.

alhafidzhamdan commented 2 years ago

I've now tried setting the temporary directory to a specific directory with permission but the error remains.


TMPDIR=/exports/eddie/scratch/mhamdan
export TEMPDIR=$TMPDIR

module load singularity

$NEOFUSE_SCRIPT \
    -1 ${FASTQ_FILE}_R1.fastq.gz \
    -2 ${FASTQ_FILE}_R2.fastq.gz \
    -s $STAR_INDEX_DIR \
    -g $STAR_GENOME \
    -a $STAR_GTF \
    -o $OUTPUT_DIR \
    -d $RUN_ID \
    -V GRCh38 \
    -n 20 \
    -m 8 \
    -M 11 \
    -c L \
    --singularity


INFO:    Convert SIF file to sandbox...
[-------------------------------- [NeoFuse] --------------------------------]

[17:35:10]  Paired End (PE) Reads detected: commencing processing
[17:35:10]  Processing files b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R1.fastq.gz - b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R2.fastq.gz
[17:35:10]  STAR Run started
[17:35:10]  Arriba Run started
[18:15:08]  YARA Run started
Couldn't create temporary file /exports/eddie/scratch/mhamdan/SQNNycLTW. (No such file or directory)
/usr/local/bin/seqan/include/seqan/file/string_mmap.h:635 FAILED!  (Memory Mapped String couldn't open temporary file)

stack trace:
  0    [0x55e72d8a055e]  seqan::ClassTest::fail() + 0xe
  1    [0x55e72d9a5824]  yara_mapper(+0x341824)
  2    [0x55e72d9eee17]  bool seqan::_remap<unsigned long, seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long>, unsigned long>(seqan::String<unsigned long, seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > >&, unsigned long) + 0x47
  3    [0x55e72d9f0504]  SeqStore<void, YaraContigsConfig<seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > > >::SeqStore() + 0x124
  4    [0x55e72d8354e9]  yara_mapper(+0x1d14e9)
  5    [0x55e72d779c15]  main + 0x5c5
  6    [0x2acb517c1bf7]  __libc_start_main + 0xe7
  7    [0x55e72d779eca]  _start + 0x2a

[18:15:10]  OptiType Run started
An error occured during OptiType run, check /exports/igmm/eddie/Glioblastoma-WGS/RNA-seq/results/NeoFuse/C3N-03473T1_primary_tumour/LOGS/C3N-03473T1_primary_tumour.optitype.log for more details
INFO:    Cleaning up image...
abyssum commented 2 years ago

The only issue that might cause this behavior I can think of right now is that singularity doesn't mount the path to the temp dir on launch. Can you try modifying your batch script to

export TEMPDIR=$OUTPUT_DIR

and tell me if that works for you.

If it doesn't work I'll try to find a workaround on Monday and maybe issue a hotfix.

alhafidzhamdan commented 2 years ago

Still the same output i'm afraid- if you don't mind having a look that would be great!

A

riederd commented 2 years ago

Hmmm, I think you should try export TEMPDIR=/mnt/out/$(echo ${OUTDIR##*/}) because within the container $OUTPUT_DIR is bind mounted to /tmp/out/.... see: https://github.com/icbi-lab/NeoFuse/blob/b71b66b058359c5b649b5147300f670f1f1f09cc/NeoFuse#L284

@abyssum we may need to explicitly bind mount a TEMPDIR into the container or automatically create one in the $OUTPUTDIR and export the corresponding TEMPDIR variable with the path within the container.

alhafidzhamdan commented 2 years ago

Hi @riederd, Still the same output:

INFO:    Convert SIF file to sandbox...
[-------------------------------- [NeoFuse] --------------------------------]

[12:33:47]  Paired End (PE) Reads detected: commencing processing
[12:33:47]  Processing files b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R1.fastq.gz - b436e69e-e5d8-4a7b-94ac-2250d9e4446a_R2.fastq.gz
[12:33:47]  STAR Run started
[12:33:47]  Arriba Run started
[13:02:51]  YARA Run started
Couldn't create temporary file /local/22344137.1.igmm_long/SQNFTqoSm. (No such file or directory)
/usr/local/bin/seqan/include/seqan/file/string_mmap.h:635 FAILED!  (Memory Mapped String couldn't open temporary file)

stack trace:
  0    [0x55ec4375055e]  seqan::ClassTest::fail() + 0xe
  1    [0x55ec43855824]  yara_mapper(+0x341824)
  2    [0x55ec4389ee17]  bool seqan::_remap<unsigned long, seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long>, unsigned long>(seqan::String<unsigned long, seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > >&, unsigned long) + 0x47
  3    [0x55ec438a0504]  SeqStore<void, YaraContigsConfig<seqan::MMap<seqan::MMapConfig<seqan::File<seqan::Async<void> >, unsigned long> > > >::SeqStore() + 0x124
  4    [0x55ec436e54e9]  yara_mapper(+0x1d14e9)
  5    [0x55ec43629c15]  main + 0x5c5
  6    [0x2b12f1bd8bf7]  __libc_start_main + 0xe7
  7    [0x55ec43629eca]  _start + 0x2a

[13:02:52]  OptiType Run started
An error occured during OptiType run, check /exports/igmm/eddie/Glioblastoma-WGS/RNA-seq/results/NeoFuse/C3N-03473T1_primary_tumour/LOGS/C3N-03473T1_primary_tumour.optitype.log for more details
INFO:    Cleaning up image...

I think perhaps it's trying to use a directory that's not been created? /local/22344137.1.igmm_long/SQNFTqoSm.

riederd commented 2 years ago

can you try export TMPDIR=/mnt/out/$(echo ${OUTDIR##*/}) I think it should be TMPDIR not TEMPDIR

abyssum commented 2 years ago

I am working on implementing this from within NeoFuse in the meantime.

alhafidzhamdan commented 2 years ago

Still the same error... thank you for looking into this!

riederd commented 2 years ago

We have a possible fix ready and it works as expected in our tests. We upload the new image soon.

abyssum commented 2 years ago

I've just released patch v1.2.2a which should fix the temp dir problem (works on our side - no need to add the $export directive on your side so you may as well remove it from your batch file). Try to update your repo and the image locally and re-run with your data. You can also remove the -V flag as this is only useful for specifying the reference version when building the indexes (see here), I would suggest modifying your call to smth along these lines:

module load singularity

$NEOFUSE_SCRIPT \
    -1 ${FASTQ_FILE}_R1.fastq.gz \
    -2 ${FASTQ_FILE}_R2.fastq.gz \
    -s $STAR_INDEX_DIR \
    -g $STAR_GENOME \
    -a $STAR_GTF \
    -o $OUTPUT_DIR \
    -d $RUN_ID \
    -n 20 \
    -m 8 \
    -M 11 \
    -c L \
    --singularity
alhafidzhamdan commented 2 years ago

This worked! Thank you very much for the quick fix!