Which references to use to assure compatibility with JAFFAL

[mdhitch]

Ask away!

I am running into an issue where by various references I am using do not appear to be compatible with the JAFFAL stage of the workflow. As such, the analysis stops with an error. I believe it is reference related, because this occurs if I use the demo data but supplying the human hg38 references I have downloaded (I have tried Encode, Ensembl and NCBI version), alas they all fail. However the chr20 references supplied within the demo data work for the demo fastqs.

Any help would be very much appreciated.

Matt

[sarahjeeeze]

Hi, do you have the error message that is output?

[mdhitch]

hi,

Would the following do?

N E X T F L O W ~ version 23.04.2 Launching/Users/MDH 1/epi2melabs/workflows/epi2me-labs/wf-transcriptomes/main.nf[inspiring_franklin] DSL2 - revision: 0b8606d864 |||||||||| _____ ____ ___ ____ __ __ _____ _ _ |||||||||| | ____| _ \_ _|___ \| \/ | ____| | | __ _| |__ ___ ||||| | _| | |_) | | __) | |\/| | _| _____| |/ _ | ' \/ | ||||| | || /| | / /| | | | ||_| | (| | |) \ \ |||||||||| |_|_| |_|___|| ||| ||\,|._/|/ |||||||||| wf-transcriptomes v1.0.0

Core Nextflow options runName : inspiring_franklin containerEngine: docker launchDir : /Users/MDH 1/epi2melabs/instances/wf-transcriptomes_01HKYZMDR881B7YQKCA6A9600W workDir : /Users/MDH 1/epi2melabs/instances/wf-transcriptomes_01HKYZMDR881B7YQKCA6A9600W/work projectDir : /Users/MDH 1/epi2melabs/workflows/epi2me-labs/wf-transcriptomes userName : MDH 1 profile : standard configFiles : /Users/MDH 1/epi2melabs/workflows/epi2me-labs/wf-transcriptomes/nextflow.config Input Options fastq : /Users/MDH 1/epi2melabs/demo/epi2me-labs/wf-transcriptomes/v1.0.0/wf-transcriptomes-demo/differential_expression_fastq ref_genome : /Volumes/MDH_Ext_HD/JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2/hg38.fa ref_annotation : /Users/MDH 1/epi2melabs/demo/epi2me-labs/wf-transcriptomes/v1.0.0/wf-transcriptomes-demo/Homo_sapiens.GRCh38.109.gtf direct_rna : true Output Options out_dir : /Users/MDH 1/epi2melabs/instances/wf-transcriptomes_01HKYZMDR881B7YQKCA6A9600W/output Sample Options sample_sheet : /Users/MDH 1/epi2melabs/demo/epi2me-labs/wf-transcriptomes/v1.0.0/wf-transcriptomes-demo/sample_sheet.csv Gene Fusion Detection Options jaffal_refBase : /Volumes/MDH_Ext_HD/JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2 Differential Expression Options de_analysis : true !! Only displaying parameters that differ from the pipeline defaults !!

If you use epi2me-labs/wf-transcriptomes for your analysis please cite:

• The nf-core framework https://doi.org/10.1038/s41587-020-0439-x

  This is epi2me-labs/wf-transcriptomes v1.0.0.

  Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files. Doing reference based transcript analysis [24/ed6621] Submitted process > pipeline:preprocess_ref_annotation [db/37de58] Submitted process > pipeline:differential_expression:checkSampleSheetCondition [09/f3c10c] Submitted process > validate_sample_sheet [5f/856c04] Submitted process > pipeline:getVersions [5f/fbd241] Submitted process > pipeline:getParams [a2/bce717] Submitted process > fastcat (4) [a8/582ca3] Submitted process > fastcat (6) [a2/b2fa72] Submitted process > fastcat (3) [7b/d687a4] Submitted process > fastcat (2) [8f/acfdb3] Submitted process > fastcat (1) [b6/f939a7] Submitted process > pipeline:collectFastqIngressResultsInDir (1) [be/be02ca] Submitted process > pipeline:collectFastqIngressResultsInDir (2) [d3/0e8fe3] Submitted process > fastcat (5) [a5/14d9df] Submitted process > pipeline:collectFastqIngressResultsInDir (3) [44/f52bb7] Submitted process > pipeline:gene_fusions:jaffal (1) [07/0ba7d9] Submitted process > pipeline:collectFastqIngressResultsInDir (4) [72/4bc4fd] Submitted process > pipeline:gene_fusions:jaffal (2) [3a/709773] Submitted process > pipeline:collectFastqIngressResultsInDir (5) [42/04bba1] Submitted process > pipeline:collectFastqIngressResultsInDir (6) [79/d30547] Submitted process > pipeline:gene_fusions:jaffal (3) ERROR ~ Error executing process > 'pipeline:gene_fusions:jaffal (2)' Caused by: Missing output file(s) jaffal_output_sample06/*jaffa_results.csv expected by process pipeline:gene_fusions:jaffal (2) Command executed: JAFFAOUT=jaffal_output_sample06

  JAFFAL exists with status code 1 when there's 0 fusion hits. Prevent this with '||:'

  /home/epi2melabs/JAFFA/tools/bin/bpipe run -n 4 -p jaffa_output="$JAFFAOUT/" -p refBase=JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2 -p genome=hg38 -p annotation=genCode22 -p fastqInputFormat="*.fastq" /home/epi2melabs/JAFFA/JAFFAL.groovy seqs.fastq.gz || :

  summary="$JAFFAOUT/all/all.summary"

  if [ -f $summary ]; then

  The summary is writtten so assume JAFFAL completed.

  if [ ! -s $summary ]; then echo "JAFFAL failed to find any fusion transcripts for sample06" touch "$JAFFAOUT/sample06_jaffa_results.csv" else echo JAFFAL found fusion transcripts for sample06 mv "$JAFFAOUT/jaffa_results.csv" "$JAFFAOUT/sample06_jaffa_results.csv"

  Add sample id column and header

    sed "s/$/,sample06/" $JAFFAOUT/sample06_jaffa_results.csv                 | sed "1 s/sample06/sample_id/" > tmp
    mv tmp $JAFFAOUT/sample06_jaffa_results.csv

  fi else echo JAFFAL encountered an error while prosessing sample06 fi Command exit status: 0 Command output: Bases Processed: 17089k 18.64m bases/sec

  ================================ Stage minimap2_transcriptome (all) ================================ [M::mm_idx_gen::11.2211.92] collected minimizers [M::mm_idx_gen::14.7732.29] sorted minimizers [M::main::14.7742.29] loaded/built the index for 195175 target sequence(s) [M::mm_mapopt_update::15.5502.22] mid_occ = 110 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 195175 [M::mm_idx_stat::15.8412.20] distinct minimizers: 16950829 (43.60% are singletons); average occurrences: 3.243; average spacing: 5.406; total length: 297165649 [M::worker_pipeline::24.3342.61] mapped 15861 sequences [M::main] Version: 2.24-r1122 [M::main] CMD: minimap2 -t 4 -x map-ont -c JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2/hg38_genCode22.fa jaffal_output_sample06/all/all.fasta [M::main] Real time: 24.385 sec; CPU: 63.464 sec; Peak RSS: 1.925 GB

  ================================== Stage filter_transcripts (all) ================================== Done reading in transcript IDs Reading the input alignment file, jaffal_output_sample06/all/all.paf 0 7182 reads processed. Finished.

  =============================== Stage extract_fusion_sequences (all) =============================== java -ea -Xmx200m -cp /home/epi2melabs/JAFFA/tools/bbmap/current/ jgi.ReformatReads in=jaffal_output_sample06/all/all.fasta out=stdout.fasta fastawrap=0 Executing jgi.ReformatReads [in=jaffal_output_sample06/all/all.fasta, out=stdout.fasta, fastawrap=0] Input is being processed as unpaired Input: 15861 reads 17089136 bases Output: 15861 reads (100.00%) 17089136 bases (100.00%) Time: 0.754 seconds. Reads Processed: 15861 21.04k reads/sec Bases Processed: 17089k 22.67m bases/sec

  =================================== Stage minimap2_genome (all) ==================================== bash: line 1: 2887 Killed minimap2 -t 4 -x splice -c JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2/hg38.fa jaffal_output_sample06/all/all.fusions.fa > jaffal_output_sample06/all/all_genome.paf ERROR: Command failed with exit status = 137 :

  minimap2 -t 4 -x splice -c JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2/hg38.fa jaffal_output_sample06/all/all.fusions.fa > jaffal_output_sample06/all/all_genome.paf; grep $'\t+\t' jaffal_output_sample06/all/all_genome.paf | awk -F'\t' -v OFS="\t" '{ print $4-$3,0,0,0,0,0,0,0,$5,$1,$2,$3,$4,$6,$7,$8,$9,2, 100","$4-$3-100",",$3","$3+100",", $8","$9-$4+$3+100"," }' > jaffal_output_sample06/all/all_genome.psl ; grep $'\t-\t' jaffal_output_sample06/all/all_genome.paf | awk -F'\t' -v OFS="\t" '{ print $4-$3,0,0,0,0,0,0,0,$5,$1,$2,$3,$4,$6,$7,$8,$9,2, 100","$4-$3-100",", $2-$4","$2-$4+100",", $8","$9-$4+$3+100"," }' >> jaffal_output_sample06/all/all_genome.psl ;

  ========================================= Pipeline Failed ==========================================

  One or more parallel stages aborted. The following messages were reported:

  Branch all in stage Unknown reported message:

  Command failed with exit status = 137 :

  minimap2 -t 4 -x splice -c JAFFA_REFERENCE_FILES_HG38_GENCODE22.V2/hg38.fa jaffal_output_sample06/all/all.fusions.fa > jaffal_output_sample06/all/all_genome.paf; grep $'\t+\t' jaffal_output_sample06/all/all_genome.paf | awk -F'\t' -v OFS="\t" '{ print $4-$3,0,0,0,0,0,0,0,$5,$1,$2,$3,$4,$6,$7,$8,$9,2, 100","$4-$3-100",",$3","$3+100",", $8","$9-$4+$3+100"," }' > jaffal_output_sample06/all/all_genome.psl ; grep $'\t-\t' jaffal_output_sample06/all/all_genome.paf | awk -F'\t' -v OFS="\t" '{ print $4-$3,0,0,0,0,0,0,0,$5,$1,$2,$3,$4,$6,$7,$8,$9,2, 100","$4-$3-100",", $2-$4","$2-$4+100",", $8","$9-$4+$3+100"," }' >> jaffal_output_sample06/all/all_genome.psl ;

  Use 'bpipe errors' to see output from failed commands.

  JAFFAL encountered an error while prosessing sample06 Command error: ln: target '/home/epi2melabs/.bpipedb/jobs/72' is not a directory Work dir: /Users/MDH 1/epi2melabs/instances/wf-transcriptomes_01HKYZMDR881B7YQKCA6A9600W/work/72/4bc4fdb81483c9423edffa33418ccd Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh -- Check '/Users/MDH 1/epi2melabs/instances/wf-transcriptomes_01HKYZMDR881B7YQKCA6A9600W/nextflow.log' file for details WARN: Killing running tasks (3)`

As a note, just running some demos locally, I initially had the issue with my own data so sought to use some example data. In the following the JAFFAL references were downloaded as suggested and I used the JAFFAL supplied reference genome. On this occasion I have used a GTF file from the EPI2ME-LABS demo directory, however I have tried very many alternatives each with the same issue. It is not a machine issue as it is repeated on a few machines I have used. I have also tried generating my own JAFFAL according to https://github.com/Oshlack/JAFFA/wiki/FAQandTroubleshooting#how-can-i-generate-the-reference-files-for-a-non-supported-genome but no luck.

The pipeline works fine when using all the demo data and the EPI2ME-LABS demo JAFFAL files. And indeed when not using Gene Fusion Detection.

Many thanks for your help and support in advance.

Matt

[nrhorner]

Hi @mdhitch

It looks like your process is running out memory. I need update the process to assign more memory to it. In the meantime you can set the memory available to the JAFFAL process by doing the following:

Create a file, lets call it mem.config, with the following contents

process {
    withName: jaffal {
        memory = 16.GB
    }
}

Then refer to this file in you comamnd like so:

`nextflow run epi2me-labs/wf-transcriptomes -c mem.config ...```

Please get back in touch if this does not work for you

Thanks,

Neil

[mdhitch]

Thanks Neil

[nrhorner]

Hi @mdhitch

Did you get a chance to try out my suggestion?

[mdhitch]

Hi @nrhorner, sorry for the belated reply, since I managed to get all outputs (minus Jaffal) I haven't yet gone back to it. Will do soon so shall attempt then, possibly with more memory than 16GB.

I will let you know how it goes.

Thanks

Matt

[sarahjeeeze]

Closing through lack of response, assuming it worked. Feel free to re-open if not.