Closed rgn3210 closed 1 year ago
skunklem
commented
1 year ago In my experience, the freyja step is less likely to fail if you allot it more memory. My runs were failing when I had larger fastq files. In startWorkflow.nf, you can try adding memory '16 GB' (or however much memory you like, since the default is 8 GB) after the line process freyjaVariantCaller {.
rgn3210
commented
1 year ago Thank you for your reply!
I have tried changing the memory to 16, 30 and 50 GB, but unfortunately I am still having the same issue (same errors). Is there anything else I could potentially try?
skunklem
commented
1 year ago If you go into the .nextflow.log file to where you're getting a line like this (where you see "exit: 1"):
May-15 17:42:29.937 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 2695446; id: 156; name: freyjaVariantCaller (4); status: COMPLETED; exit: 1; error: -; workDir: /projects/enviro_lab/WW-UNCC/NCDHHS-05-11-23-A4-1-test3-fastqs/work/57/1cc5bbc34de8fbe15786651c7c981e started: 1684186916939; exited: 2023-05-15T21:42:12.439007427Z; ]Go to that workDir and open up .command.log. Mine has a bunch of stack traces that all end with
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.That's how I knew my issue was insufficient memory. If you have a different problem going on, the solution would be different. Check out what sort of error message you're getting. It may be something with an input that you can resolve yourself. If not, post it and maybe someone can help.
rgn3210
commented
1 year ago The error message I'm getting from my .command.log files are:
Pileup generation for Freyja...
ivar: invalid option -- 'r'
[mpileup] 1 samples in 1 input files
Demixing variants by Freyja and bootstrapping
Usage: freyja demix [OPTIONS] VARIANTS DEPTHS
Try 'freyja demix --help' for help.
Error: Invalid value for 'VARIANTS': Path 'freyja.variants.tsv' does not exist.
Usage: freyja boot [OPTIONS] VARIANTS DEPTHS
Try 'freyja boot --help' for help.
Error: Invalid value for 'VARIANTS': Path 'freyja.variants.tsv' does not exist.The freyja steps are as follows:
echo Pileup generation for Freyja...
freyja variants resorted.bam --variants freyja.variants.tsv --depths freyja.depths.tsv --ref /projects/enviro_lab/software/C-WAP/covidRefSequences/wuhan.fa
echo Demixing variants by Freyja and bootstrapping
freyja demix freyja.variants.tsv freyja.depths.tsv --output freyja.demix --confirmedonly &
freyja boot freyja.variants.tsv freyja.depths.tsv --nt $(nproc) --nb 1000 --output_base freyja_boot
waitThe ivar error must be from the freyja variants step. freyja variants is what produces 'freyja.variants.tsv', so it's not getting made for some reason. Maybe there's an issue with the inputs. I would verify that the bam file exists in that workDir.
Ivar definitely has the option '-r', and the freyja command using ivar can be seen here.
In startWorkflow.sh, the command looks like this:
freyja variants resorted.bam --variants freyja.variants.tsv --depths freyja.depths.tsv --ref $params.referenceSequenceConsidering that the --ref variable is derived from the referenceSequence variable in your nextflow.config here and is the only non-explicit part of that command, maybe that variable is empty or incorrect in your config.
In $workDir/.command.sh, does the freyja variants command look correct?
Mine looks like this with the reference sequence expanded: freyja variants resorted.bam --variants freyja.variants.tsv --depths freyja.depths.tsv --ref /projects/enviro_lab/software/C-WAP/covidRefSequences/wuhan.fa
srnarum
commented
1 year ago I have the same issue and get the exact same error message from my .command.log file. The bam file does exist in my working directory and my freyja variants command looks like this:
freyja variants resorted.bam --variants freyja.variants.tsv --depths freyja.depths.tsv --ref /mnt/ceph/srnarum/C-WAP/covidRefSequences/wuhan.fa. The path to the reference file looks correct.
Did you solve this issue @rgn3210 ?
skunklem
commented
1 year ago If resorted.bam and the reference file both exist, then I'm not sure what else the issue could be (unless you have a bad version of freyja or ivar). I would recommend going into your workDir and running just the freyja variants step to make sure that works, like this.:
cd $workDir
conda activate $cwap_dir/conda/env-freyja
freyja variants resorted.bam --variants freyja.variants.tsv --depths freyja.depths.tsv --ref /mnt/ceph/srnarum/C-WAP/covidRefSequences/wuhan.faIf that works without the ivar error (ivar: invalid option -- 'r') and produces the expected outputs (variants and depths), you should be good. If not, that might at least give the developers more of an idea of the problem...
srnarum
commented
1 year ago Hm, that unfortunately does not work.
My version of ivar does not allow for a -r flag, this is the output of ivar variants:
Usage: samtools mpileup -A -d 0 --reference
Note : samtools mpileup output must be piped into ivar variants
Input Options Description -q Minimum quality score threshold to count base (Default: 20) -t Minimum frequency threshold(0 - 1) to call variants (Default: 0.03)
Output Options Description -p (Required) Prefix for the output tsv variant file
skunklem
commented
1 year ago That must be the issue. You'll need to update your version of ivar within the conda environment env-freyja. I have ivar version 1.3.1 and it has three additional Input Options (including -r). Ibelive that should be as simple as activating the env and running
conda update ivarThe current version is 1.4.2.
jasmine-amir
commented
1 year ago Hi, and thanks for choosing C-WAP!
Also, special thank you to @skunklem !!
Assuming you all installed ivar using the prepare_envs.sh, then the version should already be ivar 1.3.1, which is what we use.
We may have noticed a similar issue in-house and are discussing with the Freyja developers.
In order to determine if we are all dealing with the same issue, I am curious to see the output of a particular file, please and thank you in advance.
Here is how to find that file:
work-<hash> directory: cd work-<hash>/. This may look something like cd cd work-cdcea7b210f41b6579f3c4c60c48b5f9/, but everyone will have a slightly different hash.find . -type f -name 'freyja.variants.tsv' -print./dd/e86f512d8b17a42fa5dbc7d26e906c/freyja.variants.tsv.cd dd/e86f512d8b17a42fa5dbc7d26e906c/cat -n .command.log, and please reply with the output of that.Thanks again! Jasmine
srnarum
commented
1 year ago Hi, I did install all of the environments using prepare_envs.sh, but the ivar version is 1.0. Even after conda update ivar and rebooting, the version is still 1.0.
The output is:
Pileup generation for Freyja...
2 ivar: invalid option -- 'r'
3 [mpileup] 1 samples in 1 input files
4 Demixing variants by Freyja and bootstrapping
5 Usage: freyja demix [OPTIONS] VARIANTS DEPTHS
6 Try 'freyja demix --help' for help.
7
8 Error: Invalid value for 'VARIANTS': Path 'freyja.variants.tsv' does not exist.
9 Usage: freyja boot [OPTIONS] VARIANTS DEPTHS
10 Try 'freyja boot --help' for help.
11
12 Error: Invalid value for 'VARIANTS': Path 'freyja.variants.tsv' does not exist.
jasmine-amir
commented
1 year ago Hello,
I have updated the repo with a new release, v1.0.0, a few moments ago.
I did adjust several configs in this latest update, and I am hopeful it will resolve this issue.
Please let me know how it goes.
All the best, Jasmine
Hello!
I am trying to use your pipeline, however I am having some trouble with Freyja. Below i have included my terminal input and output.
Input:
Below I have also included a part of the output of the nextflow log:
The reports I get as the output contain all the analysis besides the Freyja part.
Could you perhaps give me some guidance on how to fix this problem? I would be really grateful!