Closed adomingues closed 7 months ago
oddly enough testing on my laptop there is a different error:
nextflow run nf-core/fetchngs -profile test,singularity -revision 1.11.0 --outdir ./test
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/fetchngs` [hopeful_perlman] DSL2 - revision: 04ee5031a4 [1.11.0]
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/fetchngs v1.11.0-g04ee503
------------------------------------------------------
Core Nextflow options
revision : 1.11.0
runName : hopeful_perlman
containerEngine : singularity
launchDir : /home/adomingues/sandbox
workDir : /home/adomingues/sandbox/work
projectDir : /home/adomingues/.nextflow/assets/nf-core/fetchngs
userName : adomingues
profile : test,singularity
configFiles :
Input/output options
input : ${projectDir}/tests/sra_ids_test.csv
outdir : ./test
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/fetchngs for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.5070524
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/fetchngs/blob/master/CITATIONS.md
------------------------------------------------------
ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
The following invalid input values have been detected:
* --input: the file or directory '${projectDir}/tests/sra_ids_test.csv' does not exist.
-- Check script '/home/adomingues/.nextflow/assets/nf-core/fetchngs/./subworkflows/local/nf_core_fetchngs_utils/../../nf-core/nfvalidation_plugin_utils/main.nf' at line: 55 or see '.nextflow.log' file for more details
OS: Ubuntu 22.04.3 LTS x86_64 Host: 21CB009UGE ThinkPad X1 Carbon Gen 10 Kernel: 6.5.0-15-generic Shell: bash 5.1.16 CPU: 12th Gen Intel i7-1255U (12) @ 4.700GHz GPU: Intel Device 46a8 Memory: 9816MiB / 15681MiB
Can you try the dev branch?
Sorry, I forgot to mention that I also tested it had the the error remains (this one the server which is ultimately where I want to test pipelines):
nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/fetchngs` [zen_borg] DSL2 - revision: 32952832ba [dev]
ERROR ~ .nextflow/plr/4bff1783-0896-407e-881e-7229e0d04070/nf-validation-1.1.3: Operation not supported
-- Check '.nextflow.log' file for details
tail -20 .nextflow.log
Feb-13 09:51:24.703 [main] ERROR nextflow.cli.Launcher - .nextflow/plr/4bff1783-0896-407e-881e-7229e0d04070/nf-validation-1.1.3: Operation not supported
java.nio.file.FileSystemException: .nextflow/plr/4bff1783-0896-407e-881e-7229e0d04070/nf-validation-1.1.3: Operation not supported
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.createSymbolicLink(UnixFileSystemProvider.java:462)
at java.base/java.nio.file.Files.createSymbolicLink(Files.java:1059)
at nextflow.plugin.LocalPluginManager.createLink0(LocalPluginManager.groovy:117)
at nextflow.plugin.LocalPluginManager.createLinkFromPath(LocalPluginManager.groovy:110)
at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:96)
at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:335)
at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:197)
at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:144)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:328)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:333)
at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:232)
at nextflow.plugin.Plugins.load(Plugins.groovy:56)
at nextflow.cli.CmdRun.run(CmdRun.groovy:334)
at nextflow.cli.Launcher.run(Launcher.groovy:500)
at nextflow.cli.Launcher.main(Launcher.groovy:672)
FYI I did another test, this time on an ec2 instance without any prior nextflow install - go / singlularity / nextflow were are all installed for the first time - and the error remains:
extflow run nf-core/fetchngs -profile test,singularity -revision 1.11.0 --input --outdir ./test
N E X T F L O W ~ version 23.10.1
Pulling nf-core/fetchngs ...
downloaded from https://github.com/nf-core/fetchngs.git
Launching `https://github.com/nf-core/fetchngs` [boring_lamport] DSL2 - revision: 04ee5031a4 [1.11.0]
Downloading plugin nf-validation@1.1.3
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/fetchngs v1.11.0-g04ee503
------------------------------------------------------
Core Nextflow options
revision : 1.11.0
runName : boring_lamport
containerEngine : singularity
launchDir : /home/ubuntu
workDir : /home/ubuntu/work
projectDir : /home/ubuntu/.nextflow/assets/nf-core/fetchngs
userName : ubuntu
profile : test,singularity
configFiles :
Input/output options
input : true
outdir : ./test
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/fetchngs for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.5070524
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/fetchngs/blob/master/CITATIONS.md
------------------------------------------------------
ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
The following invalid input values have been detected:
* --input: the file or directory 'true' does not exist.
* --input: expected type: String, found: Boolean (true)
--input
takes a file into account, so this failure is expected.
This looks like a Java version error, which version of Java are you using?
java -version
Currently we require 11 - 21, we recommend using SDKMan: https://www.nextflow.io/docs/latest/getstarted.html#requirements
--input
takes a file into account, so this failure is expected.
My bad. Without --input
:
nextflow run nf-core/fetchngs -profile test,singularity -revision 1.11.0 --outdir ./test
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/fetchngs` [mighty_goldwasser] DSL2 - revision: 04ee5031a4 [1.11.0]
WARN: Access to undefined parameter `monochromeLogs` -- Initialise it to a default value eg. `params.monochromeLogs = some_value`
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/fetchngs v1.11.0-g04ee503
------------------------------------------------------
Core Nextflow options
revision : 1.11.0
runName : mighty_goldwasser
containerEngine : singularity
launchDir : /home/ubuntu
workDir : /home/ubuntu/work
projectDir : /home/ubuntu/.nextflow/assets/nf-core/fetchngs
userName : ubuntu
profile : test,singularity
configFiles :
Input/output options
input : ${projectDir}/tests/sra_ids_test.csv
outdir : ./test
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/fetchngs for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.5070524
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/fetchngs/blob/master/CITATIONS.md
------------------------------------------------------
ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
The following invalid input values have been detected:
* --input: the file or directory '${projectDir}/tests/sra_ids_test.csv' does not exist.
-- Check script '.nextflow/assets/nf-core/fetchngs/./subworkflows/local/nf_core_fetchngs_utils/../../nf-core/nfvalidation_plugin_utils/main.nf' at line: 55 or see '.nextflow.log' file for more details
It can't find the input file so I will point directly to the file in the repo and this time the pipeline runs!
nextflow run nf-core/fetchngs -profile test,singularity -revision 1.11.0 --input https://raw.githubusercontent.com/nf-core/fetchngs/1.11.0/tests/sra_ids_test.csv --outdir ./test
-[nf-core/fetchngs] Pipeline completed successfully-
WARN: =============================================================================
Please double-check the samplesheet that has been auto-created by the pipeline.
Public databases don't reliably hold information such as strandedness
information, controls etc
All of the sample metadata obtained from the ENA has been appended
as additional columns to help you manually curate the samplesheet before
running nf-core/other pipelines.
===================================================================================
This tells me that the issue is not related to fetchngs
but rather my nextflow install. I will close the issue but appreciate any pointers on purging an nextflow / nf-core installation in the hop that a fresh install will help me.
This looks like a Java version error, which version of Java are you using?
java -version
Currently we require 11 - 21, we recommend using SDKMan: https://www.nextflow.io/docs/latest/getstarted.html#requirements
The version seems correct but openjdk:
java -version
openjdk version "11.0.21" 2023-10-17
OpenJDK Runtime Environment (build 11.0.21+9-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.21+9-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
For reference, this is the version in the ec2 ubuntu instance in which the pipeline runs successfully:
java -version
openjdk version "11.0.21" 2023-10-17
OpenJDK Runtime Environment (build 11.0.21+9-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.21+9-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
Can you try this one?
nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test
The '${projectDir}/tests/sra_ids_test.csv' does not exist
is fixed now
Can you try this one?
nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test
The nf-validation error is still there :(
nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test N E X T F L O W ~ version 23.10.1
Launchinghttps://github.com/nf-core/fetchngs
[voluminous_mandelbrot] DSL2 - revision: 32952832ba [dev] ERROR ~ .nextflow/plr/27bb8b0f-89d2-4c87-903a-726c7f161b3c/nf-validation-1.1.3: Operation not supportedcat .nextflow.log
Feb-13 12:05:25.266 [main] DEBUG nextflow.cli.Launcher - $> nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test Feb-13 12:05:25.398 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 23.10.1
Feb-13 12:05:25.442 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/mnt/genomics/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1 .3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1
Feb-13 12:05:25.467 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Feb-13 12:05:25.469 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Feb-13 12:05:25.473 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Feb-13 12:05:25.490 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Feb-13 12:05:25.514 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: /mnt/genomics/.nextflow/scm
Feb-13 12:05:26.809 [main] DEBUG nextflow.scm.AssetManager - Git config: /mnt/genomics/.nextflow/assets/nf-core/fetchngs/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/fetchng s.git
Feb-13 12:05:26.841 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Feb-13 12:05:26.858 [main] DEBUG nextflow.scm.AssetManager - Git config: /mnt/genomics/.nextflow/assets/nf-core/fetchngs/.git/config; branch: master; remote: origin; url: https://github.com/nf-core/fetchng s.git
Feb-13 12:05:27.842 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /mnt/genomics/.nextflow/assets/nf-core/fetchngs/nextflow.config
Feb-13 12:05:27.846 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /mnt/genomics/.nextflow/assets/nf-core/fetchngs/nextflow.config Feb-13 12:05:27.869 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile:test,singularity
Feb-13 12:05:29.088 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [cfc_dev, uzl_omics, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, pdc_kth, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ku_sund_dangpu, ccga_dx, crick, marv in, biohpc_gen, shifter, mana, mamba, unc_lccc, wehi, awsbatch, wustl_htcf, imperial, maestro, software_license, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester , vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, computerome, ucd_sonic, seg_globe, sanger, dkfz, pasteur, ethz_euler, test_full, imb, tuos_stanage, azurebatch, hki, crukmi, csiro_petrichor, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, self_hosted_runner, uw_hyak_pedslabs, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawul f, uod_hpc, fub_curta, uct_hpc, aws_tower, binac]
Feb-13 12:05:29.162 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Feb-13 12:05:29.163 [main] INFO nextflow.cli.CmdRun - Launchinghttps://github.com/nf-core/fetchngs
[voluminous_mandelbrot] DSL2 - revision: 32952832ba [dev]
Feb-13 12:05:29.164 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins declared=[nf-validation@1.1.3]
Feb-13 12:05:29.164 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Feb-13 12:05:29.164 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[nf-validation@1.1.3] Feb-13 12:05:29.164 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: 1.1.3
Feb-13 12:05:29.170 [main] ERROR nextflow.cli.Launcher - .nextflow/plr/27bb8b0f-89d2-4c87-903a-726c7f161b3c/nf-validation-1.1.3: Operation not supported
java.nio.file.FileSystemException: .nextflow/plr/27bb8b0f-89d2-4c87-903a-726c7f161b3c/nf-validation-1.1.3: Operation not supported
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.createSymbolicLink(UnixFileSystemProvider.java:462)
at java.base/java.nio.file.Files.createSymbolicLink(Files.java:1059) at nextflow.plugin.LocalPluginManager.createLink0(LocalPluginManager.groovy:117) at nextflow.plugin.LocalPluginManager.createLinkFromPath(LocalPluginManager.groovy:110)
at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:96)
at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:335)
at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:197)
at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:144)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:328)
at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:333)
at nextflow.plugin.PluginsFacade.load(PluginsFacade.groovy:232)
at nextflow.plugin.Plugins.load(Plugins.groovy:56)
at nextflow.cli.CmdRun.run(CmdRun.groovy:334)
at nextflow.cli.Launcher.run(Launcher.groovy:500)
at nextflow.cli.Launcher.main(Launcher.groovy:672)
But I can confirm on the clean nextflow install that '${projectDir}/tests/sra_ids_test.csv' does not exist
is fixed now
It looks like Nextflow is trying to make a symbolic link to or from the plugin directory but is failing. Here is the code that throws the error: https://github.com/nextflow-io/nextflow/blob/5e2ce9ed82ccbc70ec24a83e04f24b8d45855a78/modules/nf-commons/src/main/nextflow/plugin/LocalPluginManager.groovy#L115
Do you have permissions to write to $NXF_HOME
? By default this is at $HOME/.nextflow
. You could set NXF_HOME
or NXF_PLUGINS_DIR
to modify where the plugins are stored.
Do you have permissions to write to $NXF_HOME? By default this is at $HOME/.nextflow.
In my system it's changed to a mount point, but I have write permissions (just tested with touch $NXF_HOME/test
)
NXF_PLUGINS_DIR
is in a subdir of NXF_HOME
, for which I also have write permissions. I have set it to ~/.nf_plugins
for testing, but the issue remains.
nextflow run nf-core/fetchngs -profile test,singularity -revision dev --outdir ./test
N E X T F L O W ~ version 23.10.1
Launching `https://github.com/nf-core/fetchngs` [backstabbing_visvesvaraya] DSL2 - revision: 32952832ba [dev]
ERROR ~ .nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3: Operation not supported
-- Check '.nextflow.log' file for details
P.S.: I did change many of the default NFX paths in this system due to limited space in /home
, but until about a year ago I was running pipelines without issues. In the intervening year I have been using tower / aws so not much use until today.
Hmmm this is confusing!
OK consulted with Nextflow team, does your filesystem support symbolic links? Can you run the following to symlink the nf-validation directory to the working directory?
ln -s $NXF_HOME/plugins/nf-validation-1.1.3 .nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3
Here you go - it fails:
adomingues@vm-ad-genome:sandbox$ ls $NXF_HOME/plugins/nf-validation-1.1.3
classes lib META-INF
adomingues@vm-ad-genome:sandbox$ ls .nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3
ls: cannot access '.nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3': No such file or directory
adomingues@vm-ad-genome:sandbox$ ln -s $NXF_HOME/plugins/nf-validation-1.1.3 .nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3
ln: failed to create symbolic link '.nextflow/plr/d5f4a7bd-accd-4d3c-8033-0add9ef29a00/nf-validation-1.1.3': No such file or directory
adomingues@vm-ad-genome:sandbox$ ll .nextflow/plr/
total 0
drwxr-xr-x 2 root root 0 Feb 13 15:38 ./
drwxr-xr-x 2 root root 0 Feb 13 15:38 ../
drwxr-xr-x 2 root root 0 Mai 16 2022 empty/
OK that seems to be the problem, your filesystem doesn't support symbolic links. What filesystem are you using?
@adamrtalbot it's a cifs mount on linux. I am in touch our sys admin to get it sorted, cheers!
@adamrtalbot I can confirm that after the changes to the file system everything is working as expected. Thanks!
Description of the bug
(possibly related to #254 ?)
I ran into this error while testing the pipeline on a small server (VM):
Log file below. I initially though this was an issue with my
.nextflow/plugins
being too old so I removed it, but the error remains.Command used and terminal output
Relevant files
nextflow.log
System information
nextflow version 23.10.1.5891
local computer (VM on a server): OS: Ubuntu 20.04.6 LTS x86_64 Host: KVM RHEL 7.0.0 PC (i440FX + PIIX, 1996) Kernel: 5.4.0-170-generic Shell: bash 5.0.17 CPU: Intel Xeon (Skylake, IBRS) (32) @ 2.294GHz GPU: 00:02.0 Red Hat, Inc. QXL paravirtual graphic card Memory: 663MiB / 64302MiB