search

PGScatalog / pgsc_calc

The Polygenic Score Catalog Calculator is a nextflow pipeline for polygenic score calculation
https://pgsc-calc.readthedocs.io/en/latest/
Apache License 2.0
117 stars 21 forks source link

nf-validation cannot find the manifest path #312

Closed Carldeboer closed 4 months ago

Carldeboer commented 4 months ago

Description of the bug

Hi all I downloaded a recent nextflow via bioconda. Using the cluster's apptainer1.2.2. The error happens when running the test command: nextflow run pgscatalog/pgsc_calc -profile test,apptainer But it fails with an error shown in the example below The issue maybe related to #293 For context, I had a previous version of pgs_calc and nextflow with singularity working on this system, but a lot has changed since then.

Command used and terminal output

$ nextflow info
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
$ nextflow run pgscatalog/pgsc_calc -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Pulling pgscatalog/pgsc_calc ...
 downloaded from https://github.com/PGScatalog/pgsc_calc.git
Launching `https://github.com/pgscatalog/pgsc_calc` [mad_pesquet] DSL2 - revision: 45326ac066 [main]

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script 'my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details

Relevant files

.nextflow.log

Jun-12 15:03:58.141 [main] DEBUG nextflow.cli.Launcher - $> nextflow run pgscatalog/pgsc_calc -profile test,apptainer
Jun-12 15:03:58.366 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2
Jun-12 15:03:58.450 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=./my-pipelines2/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Jun-12 15:03:58.477 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-12 15:03:58.480 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-12 15:03:58.486 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jun-12 15:03:58.517 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-12 15:03:58.546 [main] DEBUG nextflow.scm.ProviderConfig - Using SCM config path: ./my-pipelines2/scm
Jun-12 15:03:58.601 [main] DEBUG nextflow.scm.RepositoryFactory - Found Git repository result: [RepositoryFactory]
Jun-12 15:03:58.618 [main] INFO  nextflow.cli.CmdRun - Pulling pgscatalog/pgsc_calc ...
Jun-12 15:03:58.630 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc/contents/nextflow.config
Jun-12 15:04:01.124 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc/contents/main.nf
Jun-12 15:04:01.262 [main] DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-] -> https://api.github.com/repos/pgscatalog/pgsc_calc
Jun-12 15:04:01.411 [main] DEBUG nextflow.scm.AssetManager - Pulling pgscatalog/pgsc_calc -- Using remote clone url: https://github.com/PGScatalog/pgsc_calc.git
Jun-12 15:04:06.719 [main] INFO  nextflow.cli.CmdRun -  downloaded from https://github.com/PGScatalog/pgsc_calc.git
Jun-12 15:04:06.806 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/nextflow.config
Jun-12 15:04:06.810 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/nextflow.config
Jun-12 15:04:06.829 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: ./my-pipelines2/secrets/store.json
Jun-12 15:04:06.833 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@4c635edc] - activable => nextflow.secret.LocalSecretsProvider@4c635edc
Jun-12 15:04:06.842 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,apptainer`
Jun-12 15:04:08.435 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, cfc_dev, uzl_omics, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, mccleary, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, vsc_calcua, pdc_kth, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, imperial, maestro, software_license, utd_europa, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, seg_globe, sanger, dkfz, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, azurebatch, hki, crukmi, csiro_petrichor, qmul_apocrita, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, self_hosted_runner, tufts, uw_hyak_pedslabs, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uod_hpc, fub_curta, uct_hpc, aws_tower, binac]
Jun-12 15:04:08.523 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jun-12 15:04:08.525 [main] DEBUG nextflow.cli.CmdRun - Launching `https://github.com/pgscatalog/pgsc_calc` [mad_pesquet] DSL2 - revision: 45326ac066 [main]
Jun-12 15:04:08.528 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-12 15:04:08.528 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-12 15:04:08.619 [main] DEBUG nextflow.Session - Session UUID: 2cdff23a-7844-4dc5-ae27-e79aac515d43
Jun-12 15:04:08.619 [main] DEBUG nextflow.Session - Run name: mad_pesquet
Jun-12 15:04:08.620 [main] DEBUG nextflow.Session - Executor pool size: 16
Jun-12 15:04:08.633 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-12 15:04:08.641 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-12 15:04:08.675 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 204776@login02 [206.12.119.244]
  CPUs: 16 - Mem: 187.5 GB (7.8 GB) - Swap: 0 (0)
Jun-12 15:04:08.719 [main] DEBUG nextflow.Session - Work-dir: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/work [nfs]
Jun-12 15:04:08.720 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/bin
Jun-12 15:04:08.746 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-12 15:04:08.760 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-12 15:04:08.885 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-12 15:04:08.927 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Jun-12 15:04:09.099 [main] DEBUG nextflow.Session - Session start
Jun-12 15:04:09.103 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/results/pipeline_info/execution_trace_2024-06-12_15-04-08.txt
Jun-12 15:04:09.122 [main] DEBUG nextflow.Session - Using default localLib path: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/lib
Jun-12 15:04:09.128 [main] DEBUG nextflow.Session - Adding to the classpath library: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/lib
Jun-12 15:04:09.896 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-12 15:04:11.023 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: latest
Jun-12 15:04:11.187 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-validation@1.1.3
Jun-12 15:04:11.897 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-validation-1.1.3.zip' in 'nf-validation-1.1.3'
Jun-12 15:04:11.905 [main] DEBUG nextflow.plugin.PluginUpdater - Failed atomic move for plugin /tmp/pf4j-update-downloader6398866501532599210/nf-validation-1.1.3 -> ./my-pipelines2/plugins/nf-validation-1.1.3 - Reason: /tmp/pf4j-update-downloader6398866501532599210/nf-validation-1.1.3 -> ./my-pipelines2/plugins/nf-validation-1.1.3: Invalid cross-device link - Fallback on safe move
Jun-12 15:04:12.183 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_6ee8db14b0ece2bc: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/main.nf
  Script_b4ded561a904b4b9: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf
Jun-12 15:04:12.184 [main] DEBUG nextflow.Session - Session aborted -- Cause: Cannot find the manifest path
Jun-12 15:04:12.200 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Cannot find the manifest path
        at org.pf4j.ManifestPluginDescriptorFinder.readManifestFromDirectory(ManifestPluginDescriptorFinder.java:142)
        at org.pf4j.ManifestPluginDescriptorFinder.readManifest(ManifestPluginDescriptorFinder.java:72)
        at org.pf4j.ManifestPluginDescriptorFinder.find(ManifestPluginDescriptorFinder.java:58)
        at org.pf4j.AbstractPluginManager.loadPluginFromPath(AbstractPluginManager.java:829)
        at org.pf4j.DefaultPluginManager.loadPluginFromPath(DefaultPluginManager.java:139)
        at nextflow.plugin.CustomPluginManager.loadPluginFromPath(CustomPluginManager.groovy:46)
        at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:98)
        at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:342)
        at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:198)
        at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:145)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:335)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:326)
        at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:453)
        at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:87)
        at nextflow.script.IncludeDef.loadPlugin0(IncludeDef.groovy:205)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:106)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_b4ded561a904b4b9.runScript(Script_b4ded561a904b4b9:7)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:222)
        at nextflow.script.IncludeDef.memoizedMethodPriv$loadModule0PathMapSession(IncludeDef.groovy:140)
        at nextflow.script.IncludeDef.access$0(IncludeDef.groovy)
        at nextflow.script.IncludeDef$__clinit__closure2.doCall(IncludeDef.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at groovy.lang.Closure.call(Closure.java:433)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at nextflow.script.IncludeDef.loadModule0(IncludeDef.groovy)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:112)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_6ee8db14b0ece2bc.runScript(Script_6ee8db14b0ece2bc:35)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)

System information

Nextflow version: 24.04.2 Hardware: HPC Excutor: local Container engine: Apptainer (tried singularity too, with similar results) OS: some linux pgs_calc: latest

nebfield commented 4 months ago

Your local nextflow installation is trying to pull files into an odd directory:

my-pipelines2/assets/pgscatalog/pgsc_calc/./workflows/pgsc_calc.nf'

Perhaps try manually installing:

$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
$ cd pgsc_calc
$ nextflow run main.nf -profile test,apptainer
Carldeboer commented 4 months ago

Doesn't seem to fix it:

$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
pgscatalog/pgsc_calc cloned to: pgsc_calc.5 ...
$ cd pgsc_calc/
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [sad_faraday] DSL2 - revision: c5aaae71cd

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script './workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details
$ cat .nextflow.log
Jun-13 09:20:49.846 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf -profile test,apptainer
Jun-13 09:20:49.958 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.2
Jun-13 09:20:49.994 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=~/my-pipelines/plugins; core-plugins: nf-amazon@2.5.2,nf-azure@1.6.0,nf-cloudcache@0.4.1,nf-codecommit@0.2.0,nf-console@1.1.3,nf-ga4gh@1.3.0,nf-google@1.13.2,nf-tower@1.9.1,nf-wave@1.4.2
Jun-13 09:20:50.019 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-13 09:20:50.021 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-13 09:20:50.026 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.10.0 in 'deployment' mode
Jun-13 09:20:50.045 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Jun-13 09:20:50.077 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/nextflow.config
Jun-13 09:20:50.081 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/nextflow.config
Jun-13 09:20:50.127 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: ~/my-pipelines/secrets/store.json
Jun-13 09:20:50.132 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6345e7b9] - activable => nextflow.secret.LocalSecretsProvider@6345e7b9
Jun-13 09:20:50.142 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `test,apptainer`
Jun-13 09:20:52.382 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [bih, cfc_dev, uzl_omics, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, incliva, ilifu, uge, rosalind_uge, lugh, mccleary, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, unc_longleaf, tigem, tubingen_apg, google, apollo, ipop_up, vsc_calcua, pdc_kth, googlels, daisybio, eddie, medair, biowulf, apptainer, bi, bigpurple, adcra, cedars, pawsey_setonix, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebi_codon_slurm, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, york_viking, unc_lccc, wehi, awsbatch, wustl_htcf, arcc, imperial, maestro, software_license, utd_europa, genotoul, nci_gadi, abims, janelia, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, create, sage, cambridge, jex, podman, ebi_codon, cheaha, xanadu, nyu_hpc, test, marjorie, computerome, ucd_sonic, seg_globe, sanger, dkfz, pasteur, einstein, ethz_euler, m3c, test_full, imb, ucl_cscluster, tuos_stanage, azurebatch, hki, crukmi, csiro_petrichor, qmul_apocrita, docker, engaging, gis, hypatia, psmn, eva, nygc, fgcz, conda, crg, singularity, self_hosted_runner, tufts, uw_hyak_pedslabs, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uod_hpc, fub_curta, uct_hpc, aws_tower, binac]
Jun-13 09:20:52.441 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declaration
Jun-13 09:20:52.525 [main] DEBUG nextflow.cli.CmdRun - Launching `main.nf` [sad_faraday] DSL2 - revision: c5aaae71cd
Jun-13 09:20:52.527 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-13 09:20:52.527 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-13 09:20:52.641 [main] DEBUG nextflow.Session - Session UUID: b0bfe352-a33c-4121-a56f-210b5a4efa7d
Jun-13 09:20:52.641 [main] DEBUG nextflow.Session - Run name: sad_faraday
Jun-13 09:20:52.642 [main] DEBUG nextflow.Session - Executor pool size: 16
Jun-13 09:20:52.656 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Jun-13 09:20:52.666 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-13 09:20:52.702 [main] DEBUG nextflow.cli.CmdRun -
  Version: 24.04.2 build 5914
  Created: 29-05-2024 06:19 UTC (28-05-2024 23:19 PDT)
  System: Linux 3.10.0-1160.108.1.el7.x86_64
  Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 47562@login01 [206.12.119.243]
  CPUs: 16 - Mem: 187.5 GB (10.5 GB) - Swap: 0 (0)
Jun-13 09:20:52.760 [main] DEBUG nextflow.Session - Work-dir: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/work [nfs]
Jun-13 09:20:52.810 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-13 09:20:52.828 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-13 09:20:52.941 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-13 09:20:52.966 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 17; maxThreads: 1000
Jun-13 09:20:53.166 [main] DEBUG nextflow.Session - Session start
Jun-13 09:20:53.170 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/results/pipeline_info/execution_trace_2024-06-13_09-20-52.txt
Jun-13 09:20:53.189 [main] DEBUG nextflow.Session - Using default localLib path: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/lib
Jun-13 09:20:53.195 [main] DEBUG nextflow.Session - Adding to the classpath library: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/lib
Jun-13 09:20:53.850 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-13 09:20:54.876 [main] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-validation version: latest
Jun-13 09:20:55.140 [main] INFO  nextflow.plugin.PluginUpdater - Downloading plugin nf-validation@1.1.3
Jun-13 09:20:56.068 [main] INFO  org.pf4j.util.FileUtils - Expanded plugin zip 'nf-validation-1.1.3.zip' in 'nf-validation-1.1.3'
Jun-13 09:20:56.073 [main] DEBUG nextflow.plugin.PluginUpdater - Failed atomic move for plugin /tmp/pf4j-update-downloader11363677661301068091/nf-validation-1.1.3 -> ~/my-pipelines/plugins/nf-validation-1.1.3 - Reason: /tmp/pf4j-update-downloader11363677661301068091/nf-validation-1.1.3 -> ~/my-pipelines/plugins/nf-validation-1.1.3: Invalid cross-device link - Fallback on safe move
Jun-13 09:20:56.328 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_10147630fe6ec4e7: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/./workflows/pgsc_calc.nf
  Script_6ee8db14b0ece2bc: /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgsc_calc/main.nf
Jun-13 09:20:56.328 [main] DEBUG nextflow.Session - Session aborted -- Cause: Cannot find the manifest path
Jun-13 09:20:56.340 [main] ERROR nextflow.cli.Launcher - @unknown
org.pf4j.PluginRuntimeException: Cannot find the manifest path
        at org.pf4j.ManifestPluginDescriptorFinder.readManifestFromDirectory(ManifestPluginDescriptorFinder.java:142)
        at org.pf4j.ManifestPluginDescriptorFinder.readManifest(ManifestPluginDescriptorFinder.java:72)
        at org.pf4j.ManifestPluginDescriptorFinder.find(ManifestPluginDescriptorFinder.java:58)
        at org.pf4j.AbstractPluginManager.loadPluginFromPath(AbstractPluginManager.java:829)
        at org.pf4j.DefaultPluginManager.loadPluginFromPath(DefaultPluginManager.java:139)
        at nextflow.plugin.CustomPluginManager.loadPluginFromPath(CustomPluginManager.groovy:46)
        at nextflow.plugin.LocalPluginManager.loadPluginFromPath(LocalPluginManager.groovy:98)
        at nextflow.plugin.PluginUpdater.load0(PluginUpdater.groovy:342)
        at nextflow.plugin.PluginUpdater.installPlugin(PluginUpdater.groovy:198)
        at nextflow.plugin.PluginUpdater.prepareAndStart(PluginUpdater.groovy:145)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:335)
        at nextflow.plugin.PluginsFacade.start(PluginsFacade.groovy:326)
        at nextflow.plugin.PluginsFacade.startIfMissing(PluginsFacade.groovy:453)
        at nextflow.plugin.Plugins.startIfMissing(Plugins.groovy:87)
        at nextflow.script.IncludeDef.loadPlugin0(IncludeDef.groovy:205)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:106)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_10147630fe6ec4e7.runScript(Script_10147630fe6ec4e7:7)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:222)
        at nextflow.script.IncludeDef.memoizedMethodPriv$loadModule0PathMapSession(IncludeDef.groovy:140)
        at nextflow.script.IncludeDef.access$0(IncludeDef.groovy)
        at nextflow.script.IncludeDef$__clinit__closure2.doCall(IncludeDef.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:279)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
        at groovy.lang.Closure.call(Closure.java:433)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at nextflow.script.IncludeDef.loadModule0(IncludeDef.groovy)
        at nextflow.script.IncludeDef.load0(IncludeDef.groovy:112)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at Script_6ee8db14b0ece2bc.runScript(Script_6ee8db14b0ece2bc:35)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run0(BaseScript.groovy:159)
        at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
        at nextflow.script.BaseScript.run(BaseScript.groovy:209)
        at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:236)
        at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:243)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:138)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)

Not sure if related, but our cluster appears to have masked ~ from refering to the home directory in scripts somehow. This might be something new. Consequently, directories named '~' keep appearing, which I never noticed before. Or maybe that's an apptainer thing?

$ rm pgsc_calc/ -rf      
$ nextflow clone pgscatalog/pgsc_calc -r v2.0.0-alpha.5
pgscatalog/pgsc_calc cloned to: pgsc_calc.5 ...
$ cd pgsc_calc/
$ ls
assets        conftest.py   LICENSE   nextflow.config       subworkflows
bin           docs          main.nf   nextflow_schema.json  tests
CITATIONS.md  environments  Makefile  README.md             workflows
conf          lib           modules   RELEASE-CHECKLIST.md
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [nostalgic_meninsky] DSL2 - revision: c5aaae71cd

Downloading plugin nf-validation@1.1.3
ERROR ~ Cannot find the manifest path

 -- Check script './workflows/pgsc_calc.nf' at line: 7 or see '.nextflow.log' file for more details
$ ls
'~'             conftest.py    main.nf                README.md              work
 assets         docs           Makefile               RELEASE-CHECKLIST.md   workflows
 bin            environments   modules                results
 CITATIONS.md   lib            nextflow.config        subworkflows
 conf           LICENSE        nextflow_schema.json   tests
nebfield commented 4 months ago

Thanks for the info! I've set up a new branch that is a bit more explicit about plugins. Could you please try:

$ git clone -b nf-schema https://github.com/PGScatalog/pgsc_calc.git
$ cd pgsc_calc
$ nextflow run main.nf -profile test,apptainer

Masking home is interesting. By default nextflow pulls plugins and assets into $HOME/.nextflow/.... Maybe setting an environment variable will help:

$ export NXF_HOME=/path/to/normal/dir/
$ nextflow run main.nf -profile test,apptainer
Carldeboer commented 4 months ago

Well that one seemed to work:

$ git clone -b nf-schema https://github.com/PGScatalog/pgsc_calc.git
Cloning into 'pgsc_calc'...
remote: Enumerating objects: 6202, done.
remote: Counting objects: 100% (6202/6202), done.
remote: Compressing objects: 100% (2216/2216), done.
remote: Total 6202 (delta 3629), reused 5883 (delta 3439), pack-reused 0
Receiving objects: 100% (6202/6202), 9.50 MiB | 14.99 MiB/s, done.
Resolving deltas: 100% (3629/3629), done.
Updating files: 100% (199/199), done.
$ cd pgsc_calc/
$ nextflow run main.nf -profile test,apptainer

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [amazing_neumann] DSL2 - revision: a10e219a42

Downloading plugin nf-schema@2.0.0
Downloading plugin nf-prov@1.2.2

------------------------------------------------------
  pgscatalog/pgsc_calc v2.0.0-alpha.6
------------------------------------------------------
Core Nextflow options
  runName                   : amazing_neumann
  containerEngine           : apptainer
  launchDir                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc
  workDir                   : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/work
  projectDir                : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc
  userName                  : cdeboer
  profile                   : test,apptainer
  configFiles               :

Input/output options
  input                     : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/examples/samplesheet.csv
  scorefile                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/examples/scorefiles/PGS001229_22.txt
  outdir                    : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/results

Reference options
  ref_samplesheet           : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/reference.csv
  ld_grch37                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/high-LD-regions-hg19-GRCh37.txt
  ld_grch38                 : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/high-LD-regions-hg38-GRCh38.txt
  ancestry_checksums        : /arc/project/st-cdeboer-1/cdeboer/AFib/Laksman_biobank/pgscalc_test/pgsc_calc/assets/ancestry/checksums.txt

Compatibility options
  target_build              : GRCh37

Max job request options
  max_cpus                  : 2
  max_memory                : 6.GB
  max_time                  : 6.h

Other parameters
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use pgscatalog/pgsc_calc for your analysis please cite:

* The Polygenic Score Catalog
  https://doi.org/10.1101/2024.05.29.24307783
  https://doi.org/10.1038/s41588-021-00783-5

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/pgscatalog/pgsc_calc/blob/main/CITATIONS.md

[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[-        ] PGS…IBLE:PLINK2_RELABELPVAR -
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
[-        ] PGS…Y_SCORE:SCORE_AGGREGATE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 0 of 1
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 0 of 1
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (1)
[-        ] PGS…HECK:COMBINE_SCOREFILES -
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[-        ] PGS…LC:MATCH:MATCH_VARIANTS -
[-        ] PGS…ALC:MATCH:MATCH_COMBINE -
[-        ] PGS…PPLY_SCORE:PLINK2_SCORE -
executor >  local (6)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
executor >  local (6)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (7)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
executor >  local (8)
[95/7749b5] PGS…:COMBINE_SCOREFILES (1) | 1 of 1 ✔
[-        ] PGS…TIBLE:PLINK2_RELABELBIM -
[0d/34ca39] PGS… (cineca chromosome 22) | 1 of 1 ✔
[-        ] PGS…E_COMPATIBLE:PLINK2_VCF -
[92/049bb2] PGS… (cineca chromosome 22) | 1 of 1 ✔
[46/5cbe07] PGS…:MATCH_COMBINE (cineca) | 1 of 1 ✔
[fe/bc4dab] PGS…effect type additive 0) | 1 of 1 ✔
[58/df8c62] PGS…CORE_AGGREGATE (cineca) | 1 of 1 ✔
[d3/20877d] PGS…T:SCORE_REPORT (cineca) | 1 of 1 ✔
[50/350f8c] PGS…UMPSOFTWAREVERSIONS (1) | 1 of 1 ✔
-[pgscatalog/pgsc_calc] Pipeline completed successfully-
Completed at: 14-Jun-2024 15:28:43
Duration    : 3m 50s
CPU hours   : (a few seconds)
Succeeded   : 8

And it also seems that the standard test with nextflow downloading pgsc_calc also works. Maybe my environment was messed up some how.

Thanks for your help!