Open mictadlo opened 4 months ago
Hi!
Can you check if an image was downloaded to /mnt//test/singularity/
?
Also, please use a release version:
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity
Lastly, you can also try to reduce the cores to 1 just for the test and until all Singularity images are downloaded. Sometimes there are hick-ups w/ singularity downloading in parallel:
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity --cores 1 --max_cores 1 -resume
Ah and are you executing on a laptop/workstation (-profile test,singularity,local
) or a cluster (-profile test,singularity,slurm
if it is a SLURM cluster)?
Thank you for your response. Unfortunately, it did not work:
Method 1:
> ls -hlatr singularity/
total 8.0K
-rw-rw---- 1 lor default 0 Feb 6 11:18 .nanozoo-python_rnaseq-3.8--7a7808c.img.lock
-rw-rw---- 1 lor default 0 Feb 6 11:18 .nanozoo-fastqc-0.11.8--fbfa1d7.img.lock
Method 2:
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity
> ls -hlatr singularity/
total 8.0K
-rw-rw---- 1 lor default 0 Feb 8 11:37 .nanozoo-python_rnaseq-3.8--7a7808c.img.lock
-rw-rw---- 1 lor default 0 Feb 8 11:37 .nanozoo-fastqc-0.11.8--fbfa1d7.img.lock
-rw-rw---- 1 lor default 0 Feb 8 11:37 .nanozoo-fastp-0.23.1--9f2e255.img.lock
Method 3:
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity --cores 1 --max_cores 1 -resume
ls -hlatr singularity/
total 8.0K
-rw-rw---- 1 lor default 0 Feb 8 11:42 .nanozoo-python_rnaseq-3.8--7a7808c.img.lock
-rw-rw---- 1 lor default 0 Feb 8 11:42 .nanozoo-fastqc-0.11.8--fbfa1d7.img.lock
-rw-rw---- 1 lor default 0 Feb 8 11:42 .nanozoo-fastp-0.23.1--9f2e255.img.lock
Method 4:
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity,pbspro --cores 1 --max_cores 1 -resume
Caused:
nextflow.exception.AbortOperationException: Unknown configuration profile: 'pbspro'
I usually use:
> cat $HOME/.nextflow/config
singularity {
cacheDir = '/home/lor/.nextflow/NXF_SINGULARITY_CACHEDIR'
autoMounts = true
}
process {
executor = 'pbspro'
scratch = false
cleanup = false
}
...
Hi,
yeah
nextflow run hoelzer-lab/rnaflow -r 1.4.6 -profile test,singularity,pbspro --cores 1 --max_cores 1 -resume
does not work because rnaflow does not have your
process {
executor = 'pbspro'
scratch = false
cleanup = false
}
profile activated. I added that now in a branch called pbspro
.
Could it be that you are not allowed to download/execute singularity images everywhere on your cluster? When your default is
singularity {
cacheDir = '/home/lor/.nextflow/NXF_SINGULARITY_CACHEDIR'
autoMounts = true
}
you might want to point to that directory for storing the singularity images:
nextflow pull hoelzer-lab/rnaflow # to get the recent pbspro branch
nextflow run hoelzer-lab/rnaflow -r pbspro -profile test,singularity,pbspro --cores 1 --max_cores 1 -resume --singularityCacheDir /home/lor/.nextflow/NXF_SINGULARITY_CACHEDIR
Normally, singularity is working on your machine?
Yes, I ran successfully singularity on our cluster with this Nextflow pipeline:
nextflow run nf-core/rnaseq \
-profile singularity \
-r 3.14.0 \
--input samplesheet.csv \
--fasta /work/360.genome.fasta \
--save_reference \
--gff /work/360.v103.gff.gz \
--outdir results \
--aligner star_salmon \
--trimmer fastp \
--extra_fastp_args detect_adapter_for_pe,correction \
-resume
How could I try out your branch called pbspro
?
Thank you in advance,
Michal
Hi,
nextflow pull hoelzer-lab/rnaflow # to get the recent pbspro branch
nextflow run hoelzer-lab/rnaflow -r pbspro -profile test,singularity,pbspro --cores 1 --max_cores 1 -resume --singularityCacheDir /home/lor/.nextflow/NXF_SINGULARITY_CACHEDIR
or any other location where you want to store the signularity images (--singularityCacheDir
).
Hi, Thank you, but unfortunately, it still does not work. nextflow.log.txt
Strange, somehow the pull and conversion of the singularity images fails.
Can you pull one of the images manually?
singularity pull --name nanozoo-sortmerna-2.1b--ceea1a1.img docker://nanozoo/sortmerna:2.1b--ceea1a1
If this works, you could pull the images manually (the commands are in the log you shared) and place them in the --singularityCacheDir
folder. That should work automatically... but at the moment I don't have another idea.
Thank you. Your command worked, but where should I copy the singulirty container:
> singularity pull --name nanozoo-sortmerna-2.1b--ceea1a1.img docker://nanozoo/sortmerna:2.1b--ceea1a1
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
Copying blob b389e67078a3 skipped: already exists
Copying blob 68ced04f60ab skipped: already exists
Copying blob 5d06a4773ef6 skipped: already exists
Copying blob 96cf53b3a9dd skipped: already exists
Copying blob abd6c59e6d6f skipped: already exists
Copying blob 9c388eb6d33c skipped: already exists
Copying config 6ce6de275f done
Writing manifest to image destination
Storing signatures
2024/02/14 17:48:10 info unpack layer: sha256:68ced04f60ab5c7a5f1d0b0b4e7572c5a4c8cce44866513d30d9df1a15277d6b
2024/02/14 17:48:14 info unpack layer: sha256:9c388eb6d33c40662539172f0d9a357287bd1cd171692ca5c08db2886bc527c3
2024/02/14 17:48:22 info unpack layer: sha256:96cf53b3a9dd496f4c91ab86eeadca2c8a31210c2e5c2a82badbb0dcf8c8f76b
2024/02/14 17:48:29 info unpack layer: sha256:5d06a4773ef695c9ec7a03f1475d1c833d462a51a218720c022e90a1baac8165
2024/02/14 17:48:29 info unpack layer: sha256:abd6c59e6d6f510cc801e23417db7705e82db41fde51c304a5b060e5c45c2c13
2024/02/14 17:48:29 info unpack layer: sha256:b389e67078a337f08e7944b27fad46bf72fa88c81334b4c05173b35ac22f8637
INFO: Creating SIF file...
> ls -haltr
total 136M
drwxrwx--- 3 lorencm default 4.0K Feb 5 10:26 ..
-rw-rw---- 1 lorencm default 410 Feb 14 16:30 nfcorRNAseq_pbs.sh
drwxrwx--- 2 lorencm default 4.0K Feb 14 16:31 work
drwxrwx--- 3 lorencm default 4.0K Feb 14 16:32 results
-rw-rw---- 1 lorencm default 51K Feb 14 16:32 .nextflow.log
drwxrwx--- 4 lorencm default 4.0K Feb 14 16:32 .nextflow
-rwxrwx--- 1 lorencm default 136M Feb 14 17:51 nanozoo-sortmerna-2.1b--ceea1a1.img
drwxrwx--- 5 lorencm default 4.0K Feb 15 11:16 .
Thank you, Michal
Ok, great, this works. The image must be placed in the folder you are pointing to via --singularityCacheDir
in the nextflow run command. It might work if you could download all the necessary images like that, place them in the folder, and then run the pipeline.
But it's still strange then that the auto-download fails.
Hi, I was able to download all Singularity containers manually with thouse commands:
singularity pull --name nanozoo-sortmerna-2.1b--ceea1a1.img docker://nanozoo/sortmerna:2.1b--ceea1a1
singularity pull --name nanozoo-hisat2:2.1.0--66dae66.img docker://nanozoo/hisat2:2.1.0--66dae66
singularity pull --name nanozoo-minimap2:2.22--b178b67.img docker://nanozoo/minimap2:2.22--b178b67
singularity pull --name nanozoo-python_rnaseq:3--23823db.img docker://nanozoo/python_rnaseq:3--23823db
singularity pull --name nanozoo-deseq2:1.34.0--c670fa0.img docker://nanozoo/deseq2:1.34.0--c670fa0
singularity pull --name nanozoo-webgestalt:0.4.6--5a8d95e.img docker://nanozoo/webgestalt:0.4.6--5a8d95e
singularity pull --name nanozoo-fastp:0.23.1--9f2e255.img docker://nanozoo/fastp:0.23.1--9f2e255
singularity pull --name nanozoo-fastqc:0.11.8--fbfa1d7.img docker://nanozoo/fastqc:0.11.8--fbfa1d7
singularity pull --name nanozoo-subread:2.0.1--713a8e7.img docker://nanozoo/subread:2.0.1--713a8e7
singularity pull --name nanozoo-multiqc:1.12--4f89fda.img docker://nanozoo/multiqc:1.12--4f89fda
singularity pull --name nanozoo-nanoplot:1.38.1--e303519.img docker://nanozoo/nanoplot:1.38.1--e303519
I reran your workflow with
nextflow pull hoelzer-lab/rnaflow # to get the recent pbspro branch
nextflow run hoelzer-lab/rnaflow -r pbspro -profile test,singularity,pbspro --cores 1 --max_cores 1 -resume --singularityCacheDir /home/lorencm/.nextflow/NXF_SINGULARITY_CACHEDIR
Unfortunately, I got this error message:
Feb-20 11:24:33.776 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=expression_reference_based:deseq2 (1); work-dir=/mnt/hpccs01/scratch/waterhouse_team/chamilka/qld/test/work/6e/50d0b65389b7645a4f332204afd346
error [nextflow.exception.ProcessFailedException]: Process `expression_reference_based:deseq2 (1)` terminated with an error exit status (137)
Please find attached the log file nextflow.log.txt.
What did I do wrong?
Michal
Hi!
This looks like an out of memory (RAM) error at the deseq2 step. Does the pipeline finish successfully some of the previous steps?
You can also remove the cores and max_cores parameters now. The resources per process are handled by the nodes.config file which should be used when you activate the new pbspro profile.
https://github.com/hoelzer-lab/rnaflow/blob/pbspro/configs/nodes.config#L3
Maybe there is something special about the pbspro scheduler that I am not aware of and can't test.
For debugging, you could try to run the same command but instead of pbspro use local. If possible, manually connect to a computer node first. But then you should also use the parameters cores and max_cores and increase them to something like 4 and 8, respectively. By that we can somewhat ignore the pbspro scheduler and see if the problem is somehow related to this.
Lastly, can you check the work dir folder path that is written in your error command above and look for the hidden .nextflow.log file of the process that fails? And please share it?
Thx for all the debugging and sorry that it does not run out of the box!
Hi, Thank you. I tried to override the settings with the below file:
> cat nextflow.config
process {
withName: 'deseq2' {
memory = 60 GB
CPU = 24
time = 15 h
}
}
Unfortuntely, I got this error:
Feb-21 09:58:43.188 [main] ERROR nextflow.cli.Launcher - Unable to parse config file: '/mnt/hpccs01/scratch/waterhouse_team/chamilka/qld/test/nextflow.config'
No signature of method: java.lang.Integer.call() is applicable for argument types: (ConfigObject) values: [[:]]
Possible solutions: wait(), any(), abs(), wait(long), tap(groovy.lang.Closure), any(groovy.lang.Closure)
groovy.lang.MissingMethodException: No signature of method: java.lang.Integer.call() is applicable for argument types: (ConfigObject) values: [[:]]
Possible solutions: wait(), any(), abs(), wait(long), tap(groovy.lang.Closure), any(groovy.lang.Closure)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:70)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
at _nf_config_91a5c13c$_run_closure1$_closure2.doCall(_nf_config_91a5c13c:3)
at _nf_config_91a5c13c$_run_closure1$_closure2.doCall(_nf_config_91a5c13c)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:38)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
at nextflow.config.ConfigParser$_parse_closure5.doCall(ConfigParser.groovy:427)
at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:83)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1118)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:814)
at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:140)
at groovy.lang.GroovyObject.invokeMethod(GroovyObject.java:39)
at groovy.lang.Script.invokeMethod(Script.java:96)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:408)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:348)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at _nf_config_91a5c13c$_run_closure1.doCall(_nf_config_91a5c13c:2)
at _nf_config_91a5c13c$_run_closure1.doCall(_nf_config_91a5c13c)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:38)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:53)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
at nextflow.config.ConfigParser$_parse_closure5.doCall(ConfigParser.groovy:427)
at jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:83)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1118)
Please find attached the log file ( nextflow.log.txt )
What did I miss?
Michal
Hey, you used this small nextflow.config file with nothing more than the deseq2 config? This will not work, because the original nextflow.config has much more necessary information.
Please try the other points I mentioned above. I hope we can narrow the problem down somehow.
Thanks!
Hi, I have got an error running the test with singularity, as shown below:
What did I miss?
Thank you in advance,
Michal