Closed bc2zb closed 6 years ago
apeltzer
commented
6 years ago Can you try upgrading Nextflow already to latest? The way we specify containers (without the docker:// prefix for example) is something that was just recently introduced and could already break things.
I'll have a look at this soon too...
bc2zb
commented
6 years ago Unfortunately, I cannot update or change the install of nextflow because of permissions. I can reach out to the HPC support staff to see if they are willing to update nextflow though. If it wasn't clear, I have already downloaded the singularity image, so I don't think that is causing the issue.
ewels
commented
6 years ago Note that it's pretty unusual for users to not be able to install nextflow on their own user account. This is what we do on all HPC systems I've used - there's no need for admin permissions or anything to have nextflow installed. Just use the command on the nextflow homepage and then move the nextflow binary to somewhere on your PATH.
Minor thing: your command line -with-singularity '/../../nfcore-chipseq.img' \ has a leading /, so is an absolute path. Not sure if this is just from the command sanitation though.
As you say, the log Applying config profile: 'standard' is strange if you're supplying another profile name. I've never seen this happen before (even with older versions of nextflow).
And finally, yes - the error message makes no sense, irrespective of all of the above. There's no reason that the variable should be empty when it's clearly defined in the code. I suspect that there must be some minor seemingly irrelevant error somewhere. Is your altered code available anywhere we can see it?
Phil
bc2zb
commented
6 years ago Ah, I misread something on the documentation and didn't realize I could update nextflow. Ok then, I did that and the profile error is gone now. I'm now getting an access denied error, which is probably just a permissions issue. I wonder if using my user installed nextflow might be causing that? I do have permissions to the directory. Should I not be using absolute paths? Am I being a little too paranoid about directories?
ewels
commented
6 years ago Ok great! We're making progress :) What's the access denied error?
Absolute paths by themselves are fine, but /../../ does not look like a valid path...
bc2zb
commented
6 years ago Yeah, that's just because they were sanitized. Error is below. It looks similar to what you got here.
Jul-11 11:25:20.891 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 0.30.2
Jul-11 11:25:22.082 [main] INFO nextflow.cli.CmdRun - Launching `../nf-core/chipseq-master/main.nf` [mad_spence] - revision: b11db350eb
Jul-11 11:25:22.105 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: ../nf-core/chipseq-master/nextflow.config
Jul-11 11:25:22.108 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/capaldobj/nf-core/chipseq-master/nextflow.config
Jul-11 11:25:22.123 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `my_profile`
Jul-11 11:25:22.773 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, uppmax_devel, my_profile, test, conda, singularity, uppmax, none, aws, uppmax_modules, docker]
Jul-11 11:25:22.805 [main] DEBUG nextflow.config.ConfigBuilder - Enabling execution in Singularity container as requested by cli option `-with-singularity null`
Jul-11 11:25:22.844 [main] DEBUG nextflow.Session - Session uuid: a6a17cac-b673-4468-8a21-e43b72315cd9
Jul-11 11:25:22.844 [main] DEBUG nextflow.Session - Run name: mad_spence
Jul-11 11:25:22.847 [main] DEBUG nextflow.Session - Executor pool size: 2
Jul-11 11:25:22.870 [main] DEBUG nextflow.cli.CmdRun -
Version: 0.30.2 build 4867
Modified: 16-06-2018 17:49 UTC (13:49 EDT)
System: Linux 3.10.0-693.2.2.el7.x86_64
Runtime: Groovy 2.4.15 on OpenJDK 64-Bit Server VM 1.8.0_144-b01
Encoding: UTF-8 (UTF-8)
Process: 56652@cn3350 [10.2.9.178]
CPUs: 2 - Mem: 251.6 GB (112.8 GB) - Swap: 2 GB (2 GB)
Jul-11 11:25:22.924 [main] DEBUG nextflow.Session - Work-dir: ../nf-core-chip-seq/HH3C3BGX5-sampleid/work [gpfs]
Jul-11 11:25:23.481 [main] DEBUG nextflow.Session - Session start invoked
Jul-11 11:25:23.487 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Jul-11 11:25:23.488 [main] DEBUG nextflow.trace.TraceFileObserver - Flow starting -- trace file: ../nf-core-chip-seq/HH3C3BGX5-sampleid/pipeline_info/nfcore-chipseq_trace.txt
Jul-11 11:25:23.491 [main] ERROR nextflow.cli.Launcher - /../path/to/results/dir/
java.nio.file.AccessDeniedException: /../path/to/results/dir/
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
at nextflow.trace.TraceFileObserver.onFlowStart(TraceFileObserver.groovy:202)
at nextflow.Session$_start_closure14.doCall(Session.groovy:458)
at nextflow.Session$_start_closure14.call(Session.groovy)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2125)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2110)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2151)
at nextflow.Session.start(Session.groovy:458)
at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:158)
at nextflow.cli.CmdRun.run(CmdRun.groovy:232)
at nextflow.cli.Launcher.run(Launcher.groovy:428)
at nextflow.cli.Launcher.main(Launcher.groovy:586)Profiles section from my nextflow.config file (only thing I altered there)
profiles {
standard {
includeConfig 'conf/base.config'
}
conda { process.conda = "$baseDir/environment.yml" }
docker { docker.enabled = true }
singularity { singularity.enabled = true }
uppmax {
includeConfig 'conf/base.config'
includeConfig 'conf/biowulf.config'
}
uppmax_modules {
includeConfig 'conf/base.config'
includeConfig 'conf/uppmax.config'
includeConfig 'conf/uppmax-modules.config'
includeConfig 'conf/igenomes.config'
}
uppmax_devel {
includeConfig 'conf/base.config'
includeConfig 'conf/uppmax.config'
includeConfig 'conf/uppmax-devel.config'
includeConfig 'conf/igenomes.config'
}
test {
includeConfig 'conf/base.config'
includeConfig 'conf/test.config'
includeConfig 'conf/igenomes.config'
}
aws {
includeConfig 'conf/base.config'
includeConfig 'conf/aws.config'
includeConfig 'conf/igenomes.config'
}
my_profile {
includeConfig 'conf/base.config'
includeConfig 'conf/my_profile.config'
}
none {
// Don't load any config (for use with custom home configs)
}
}And the contents of my_profile.config. I just copied the uppmax profile and removed the clusterOptions (as I didn't think they'd be necessary).
/*
* -------------------------------------------------
* Nextflow config file for UPPMAX (milou / irma)
* -------------------------------------------------
* Imported under the 'uppmax' Nextflow
* profile in nextflow.config
*/
singularity {
enabled = true
}
process {
executor = 'slurm'
}
params {
max_memory = 128.GB
max_cpus = 16
max_time = 240.h
}I guess /../path/to/results/dir/ is sanitised too..? Does this exist and do you have permissions? It looks like a core java error. The issue you linked to is resolved now, so shouldn't happen again with the version you're running (Nextflow will create the output directory if it doesn't already exist).
bc2zb
commented
6 years ago Yes, it's sanitized. It's just a really long path. This issue is resolved, so I'm closing it now. I do indeed have permissions. I'll poke around the java stuff. Thanks for the assistance.
ewels
commented
6 years ago Ok, sounds good. Thanks! Let us know if there's anything we can help with, or if the solution is something that could help anyone else 👍
Let me start off by saying I'm using Nextflow 0.28.2, it's what is available on my HPC environment. If that's the issue, then so be it. I'm attempting to use the chipseq pipeline on a slurm cluster with singularity. The command I'm running (paths have been anonymized, but they are correct):
This returns:
And the log file:
No matter what I do with the nextflow.config or with the -profile option, nextflow always runs the standard profile. I've even replaced the config in the standard profile with my configs, and it still doesn't work. Additionally, I changed the container param to point to my local singularity image. Am I doing something wrong in terms of specifying a profile?