search

PGScatalog / pgsc_calc

The Polygenic Score Catalog Calculator is a nextflow pipeline for polygenic score calculation
https://pgsc-calc.readthedocs.io/en/latest/
Apache License 2.0
121 stars 21 forks source link

v2.0.0-beta.3 run: failed to find loop device, then --resume results in Failed to invoke observer completion handler (NullPointerException in BcoRenderer) #362

Open Fiwx opened 3 months ago

Fiwx commented 3 months ago

Description of the bug

I ran the current version of pgsc_calc with 63 scorefiles on a single sample and got this error:

  FATAL:   container creation failed: mount /proc/self/fd/4->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/4: failed to find loop device: could not attach image file to loop device: no loop devices available

I re-ran pgsc_calc with the --resume flag, and it seemed to work (correctly generated the _pgs.txt.gz file), but also it had this strange error:

Aug-22 20:35:43.569 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Aug-22 20:35:43.819 [main] DEBUG nextflow.Session - Failed to invoke observer completion handler: nextflow.prov.ProvObserver@101f2ca2
java.lang.NullPointerException: null
        at nextflow.prov.BcoRenderer$_render_closure3.doCall(BcoRenderer.groovy:93)
        at nextflow.prov.BcoRenderer$_render_closure3.call(BcoRenderer.groovy)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3661)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3646)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3692)
        at nextflow.prov.BcoRenderer.render(BcoRenderer.groovy:91)
        at nextflow.prov.ProvObserver$_onFlowComplete_closure4.doCall(ProvObserver.groovy:111)
        at nextflow.prov.ProvObserver$_onFlowComplete_closure4.call(ProvObserver.groovy)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2394)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2379)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2420)
        at nextflow.prov.ProvObserver.onFlowComplete(ProvObserver.groovy:110)
        at nextflow.Session.notifyFlowComplete(Session.groovy:1115)
        at nextflow.Session.shutdown0(Session.groovy:738)
        at nextflow.Session.destroy(Session.groovy:683)
        at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:260)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:146)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)
Aug-22 20:35:43.822 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Aug-22 20:35:43.841 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-prov@1.2.2'
Aug-22 20:35:43.841 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-prov
Aug-22 20:35:43.841 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-schema@2.0.0'
Aug-22 20:35:43.841 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema
Aug-22 20:35:43.841 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Aug-22 20:35:43.843 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

It says there was an error when trying to render the BioCompute Object (BCO), as indicated by the NullPointerException in the BcoRenderer, which I haven't heard of.

Do you have any ideas about what could be causing this? It is odd that resuming with no other changes works, but it doesn't work on its own the first try. I replicated this on another machine with a different sample file.

Command used and terminal output

First run:

Aug-22 18:04:56.799 [main] DEBUG nextflow.cli.Launcher - $> nextflow run /home/linux/urunner/tools/pgsc_calc/main.nf -profile singularity --input /home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e_output/samplesheet.csv --scorefile '/home/linux/urunner/system/test/custom_scores/*.txt.gz' --target_build GRCh37 --min_overlap 0.0 --pca_maf_target 0.0 --run_ancestry /home/linux/urunner/data/pgsc_1000G_v1.tar.zst -c /home/linux/urunner/references/custom.config --genotypes_cache /home/linux/urunner/system/test/cache
Aug-22 18:04:56.877 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.4

[...]

Aug-22 18:05:04.568 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Aug-22 18:05:04.568 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Aug-22 18:05:04.588 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:pyyaml` matches labels `pyyaml` for process with name PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS
Aug-22 18:05:04.589 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:DUMPSOFTWAREVERSIONS` matches process PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS
Aug-22 18:05:04.590 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Aug-22 18:05:04.590 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Aug-22 18:05:04.593 [main] DEBUG nextflow.Session - Config process names validation disabled as requested
Aug-22 18:05:04.594 [main] DEBUG nextflow.Session - Igniting dataflow network (51)
Aug-22 18:05:04.603 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /home/linux/urunner/system/test/custom_scores/;pattern: *.txt.gz; options: [checkIfExists:true]
Aug-22 18:05:04.604 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES
Aug-22 18:05:04.605 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM
Aug-22 18:05:04.605 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR
Aug-22 18:05:04.606 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF
Aug-22 18:05:04.608 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE
Aug-22 18:05:04.609 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS
Aug-22 18:05:04.610 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FILTER_VARIANTS
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_REF
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_THINNED
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:RELABEL_IDS
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_TARGET
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_ORIENT
Aug-22 18:05:04.611 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PCA
Aug-22 18:05:04.612 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PROJECT
Aug-22 18:05:04.612 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS
Aug-22 18:05:04.612 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE
Aug-22 18:05:04.612 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:RELABEL_SCOREFILES
Aug-22 18:05:04.612 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:RELABEL_AFREQ
Aug-22 18:05:04.613 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:PLINK2_SCORE
Aug-22 18:05:04.613 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:SCORE_AGGREGATE
Aug-22 18:05:04.616 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:ANCESTRY_ANALYSIS
Aug-22 18:05:04.617 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT
Aug-22 18:05:04.617 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS
Aug-22 18:05:04.618 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_6c7f1975e3fb3b69: /home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/ancestry/../../../modules/local/ancestry/oadp/fraposa_pca.nf
  Script_f7e6cf6aaa57ea1f: /home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/../../modules/local/ancestry/ancestry_analysis.nf
  Script_b30368b094794ddb: 

[...]

/home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/../../modules/local/ancestry/relabel_scorefiles.nf
  Script_d992248657715d07: /home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/match.nf
  Script_610f3cd49143490f: /home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/ancestry/../../../modules/local/ancestry/intersect_variants.nf
  Script_9484cf4d32ea80d5: /home/linux/urunner/tools/pgsc_calc/./workflows/../subworkflows/local/ancestry/../../../modules/local/ancestry/relabel_ids.nf
Aug-22 18:05:04.618 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Aug-22 18:05:04.618 [main] DEBUG nextflow.Session - Session await
Aug-22 18:05:04.702 [Actor Thread 15] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=oras://ghcr.io/pgscatalog/pygscatalog:pgscatalog-utils-1.3.1-singularity; path=/home/linux/singularity_containers/ghcr.io-pgscatalog-pygscatalog-pgscatalog-utils-1.3.1-singularity.img
Aug-22 18:05:04.702 [Actor Thread 12] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=oras://ghcr.io/pgscatalog/plink2:2.00a5.10-singularity; path=/home/linux/singularity_containers/ghcr.io-pgscatalog-plink2-2.00a5.10-singularity.img
Aug-22 18:05:04.702 [Actor Thread 1] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=oras://ghcr.io/pgscatalog/zstd:2-beta-singularity; path=/home/linux/singularity_containers/ghcr.io-pgscatalog-zstd-2-beta-singularity.img
Aug-22 18:05:04.785 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Aug-22 18:05:04.787 [Task submitter] INFO  nextflow.Session - [c3/d6f05b] Submitted process > PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE(1)
Aug-22 18:05:04.798 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Aug-22 18:05:04.799 [Task submitter] INFO  nextflow.Session - [4c/0511f6] Submitted process > PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (file10000file8266yearofsampleunknownsysknown chromosome ALL)
Aug-22 18:05:04.817 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Aug-22 18:05:04.817 [Task submitter] INFO  nextflow.Session - [44/36bba0] Submitted process > PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)
Aug-22 18:05:05.350 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1); status: COMPLETED; exit: 255; error: -; workDir: /home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e/work/44/36bba0dac829675c5a564fefc1fbe5]
Aug-22 18:05:05.351 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Aug-22 18:05:05.356 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1); work-dir=/home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e/work/44/36bba0dac829675c5a564fefc1fbe5
  error [nextflow.exception.ProcessFailedException]: Process `PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)` terminated with an error exitstatus (255)
Aug-22 18:05:05.374 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)'

Caused by:
  Process `PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)` terminated with an error exit status (255)

Command executed:

  pgscatalog-combine -s [many correct PGS ID .txt.gz scorefiles here]             -t GRCh37             -oscorefiles.txt.gz             -l log_scorefiles.json             -v             -v

  cat <<-END_VERSIONS > versions.yml
  COMBINE_SCOREFILES:
      pgscatalog.core: $(echo $(python -c 'import pgscatalog.core; print(pgscatalog.core.__version__)'))
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   container creation failed: mount /proc/self/fd/4->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/4: failed to find loop device: could not attach image file to loop device: no loop devices available

Work dir:
  /home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e/work/44/36bba0dac829675c5a564fefc1fbe5

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`
Aug-22 18:05:05.404 [TaskFinalizer-1] INFO  nextflow.Session - Execution cancelled -- Finishing pending tasks before exit
Aug-22 18:05:05.409 [main] DEBUG nextflow.Session - Session await > all processes finished
Aug-22 18:05:05.452 [Actor Thread 11] DEBUG nextflow.file.SortFileCollector - FileCollector temp dir not removed: null
Aug-22 18:05:34.964 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1); status: COMPLETED; exit: 0; error: -; workDir: /home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e/work/c3/d6f05bd46f0f66cb65423e52837709]
Aug-22 18:05:35.056 [TaskFinalizer-2] DEBUG nextflow.processor.TaskProcessor - Process PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE > Skipping output binding because one or more optional files are missing: fileoutparam<0:3>
Aug-22 18:05:35.057 [TaskFinalizer-2] DEBUG nextflow.processor.TaskProcessor - Process PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE > Skipping output binding because one or more optional files are missing: fileoutparam<1:1>
Aug-22 18:05:52.365 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (file10000file8266yearofsampleunknownsysknown chromosome ALL); status: COMPLETED; exit: 0; error: -; workDir: /home/linux/urunner/system/test/file10000_file8266_yearofsample_unknown_sex_unknown.e/work/4c/0511f6bd4eb5655ffc4393cb213f1b]
Aug-22 18:05:52.366 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Aug-22 18:05:52.366 [main] DEBUG nextflow.Session - Session await > all barriers passed

Run with resume:

Aug-22 18:47:32.673 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 24.04.4

[...]

Aug-22 18:47:34.483 [main] DEBUG nextflow.Session - Work-dir: /home/linux/urunner/system/test/work [overlayfs]
Aug-22 18:47:34.483 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/linux/urunner/tools/pgsc_calc/bin
Aug-22 18:47:34.494 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Aug-22 18:47:34.504 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Aug-22 18:47:34.526 [main] DEBUG nextflow.Session - Observer factory: ProvObserverFactory
Aug-22 18:47:34.562 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Aug-22 18:47:34.571 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Aug-22 18:47:34.665 [main] DEBUG nextflow.Session - Session start
Aug-22 18:47:34.668 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /home/linux/urunner/system/test/results/pipeline_info/execution_trace_2024-08-22_18-47-34.txt
Aug-22 18:47:34.675 [main] DEBUG nextflow.Session - Using default localLib path: /home/linux/urunner/tools/pgsc_calc/lib
Aug-22 18:47:34.679 [main] DEBUG nextflow.Session - Adding to the classpath library: /home/linux/urunner/tools/pgsc_calc/lib
Aug-22 18:47:35.083 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Aug-22 18:47:35.093 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [paramsHelp:paramsHelp]; plugin Id: nf-schema
Aug-22 18:47:35.525 [main] DEBUG nextflow.script.IncludeDef - Loading included plugin extensions with names: [validateParameters:validateParameters, paramsSummaryLog:paramsSummaryLog, paramsSummaryMap:paramsSummaryMap]; plugin Id: nf-schema

[...]

Singularity found local store for image=oras://ghcr.io/pgscatalog/pyyaml:6.0-singularity; path=/home/linux/singularity_containers/ghcr.io-pgscatalog-pyyaml-6.0-singularity.img
Aug-22 20:34:35.895 [Actor Thread 29] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-5443158211791858714
Aug-22 20:34:35.915 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Aug-22 20:34:35.915 [Task submitter] INFO  nextflow.Session - [44/e7b285] Submitted process > PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS (1)
Aug-22 20:34:36.235 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 24; name: PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS (1); status: COMPLETED; exit: 0; error: -; workDir: /home/linux/urunner/system/test/work/44/e7b285c26199eb78ed76fb94b7ce3f]
Aug-22 20:35:31.647 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 23; name: PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:ANCESTRY_ANALYSIS (1); status: COMPLETED; exit: 0; error: -; workDir: /home/linux/urunner/system/test/work/1d/aa8580a8605543ae3efec74216f511]
Aug-22 20:35:31.662 [Actor Thread 20] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=oras://ghcr.io/pgscatalog/report:2-beta-singularity; path=/home/linux/singularity_containers/ghcr.io-pgscatalog-report-2-beta-singularity.img
Aug-22 20:35:31.688 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Aug-22 20:35:31.688 [Task submitter] INFO  nextflow.Session - [66/034f9e] Submitted process > PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT (file10000file8266yearofsampleunknownsysknown)
Aug-22 20:35:42.899 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 25; name: PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT (file10000file8266yearofsampleunknownsysknown); status: COMPLETED; exit: 0; error: -; workDir: /home/linux/urunner/system/test/work/66/034f9eea344153c94f1b87f6f89382]
Aug-22 20:35:42.905 [main] DEBUG nextflow.Session - Session await > all processes finished
Aug-22 20:35:43.000 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Aug-22 20:35:43.000 [main] DEBUG nextflow.Session - Session await > all barriers passed
Aug-22 20:35:43.005 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Aug-22 20:35:43.006 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Aug-22 20:35:43.022 [main] INFO  nextflow.Nextflow - -[pgscatalog/pgsc_calc] Pipeline completed successfully-
Aug-22 20:35:43.027 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=23; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=4h 41m 28s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=8; peakMemory=56 GB; ]
Aug-22 20:35:43.027 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Aug-22 20:35:43.028 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Aug-22 20:35:43.569 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Aug-22 20:35:43.819 [main] DEBUG nextflow.Session - Failed to invoke observer completion handler: nextflow.prov.ProvObserver@101f2ca2
java.lang.NullPointerException: null
        at nextflow.prov.BcoRenderer$_render_closure3.doCall(BcoRenderer.groovy:93)
        at nextflow.prov.BcoRenderer$_render_closure3.call(BcoRenderer.groovy)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3661)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3646)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3692)
        at nextflow.prov.BcoRenderer.render(BcoRenderer.groovy:91)
        at nextflow.prov.ProvObserver$_onFlowComplete_closure4.doCall(ProvObserver.groovy:111)
        at nextflow.prov.ProvObserver$_onFlowComplete_closure4.call(ProvObserver.groovy)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2394)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2379)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2420)
        at nextflow.prov.ProvObserver.onFlowComplete(ProvObserver.groovy:110)
        at nextflow.Session.notifyFlowComplete(Session.groovy:1115)
        at nextflow.Session.shutdown0(Session.groovy:738)
        at nextflow.Session.destroy(Session.groovy:683)
        at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:260)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:146)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
        at nextflow.cli.Launcher.run(Launcher.groovy:503)
        at nextflow.cli.Launcher.main(Launcher.groovy:657)
Aug-22 20:35:43.822 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Aug-22 20:35:43.841 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-prov@1.2.2'
Aug-22 20:35:43.841 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-prov
Aug-22 20:35:43.841 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-schema@2.0.0'
Aug-22 20:35:43.841 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-schema
Aug-22 20:35:43.841 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Aug-22 20:35:43.843 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Relevant files

No response

System information

Nextflow version: 24.04.4 build 5917 System: Linux 6.5.0-1022-aws Runtime: Groovy 4.0.21 on OpenJDK 64-Bit Server VM 11.0.24+8-post-linux-1linux322.04 Encoding: UTF-8 (ANSI_X3.4-1968) CPUs: 8 - Mem: 62.1 GB (8.9 GB) - Swap: 0 (0)

"/home/linux/urunner/system/test/file10001_filestudy_yearofsample_unknown_sex_unknown.e_output/samplesheet.csv", "format": "csv", "scorefile": "/home/linux/urunner/system/test/custom_scores/*.txt.gz", "pgs_id": null, "trait_efo": null, "efo_id": null, "pgp_id": null, "efo_direct": false, "run_ancestry": "/home/linux/urunner/data/pgsc_1000G_v1.tar.zst", "ancestry_checksums": "/home/linux/urunner/tools/pgsc_calc/assets/ancestry/checksums.txt", "hg19_chain": null, "hg38_chain": null, "load_afreq": true, "geno_ref": 0.1, "mind_ref": 0.1, "maf_ref": 0.05, "hwe_ref": 0.0001, "indep_pairwise_ref": "1000 50 0.05", "ld_grch37": "/home/linux/urunner/tools/pgsc_calc/assets/ancestry/high-LD-regions-hg19-GRCh37.txt", "ld_grch38": "/home/linux/urunner/tools/pgsc_calc/assets/ancestry/high-LD-regions-hg38-GRCh38.txt", "ref_format_version": "v0.1", "ref_samplesheet": "/home/linux/urunner/tools/pgsc_calc/assets/ancestry/reference.csv", "projection_method": "oadp", "ancestry_method": "RandomForest", "ref_label": "SuperPop", "n_popcomp": 5, "normalization_method": "empirical mean mean+var", "n_normalization": 4, "pca_maf_target": 0.0, "pca_geno_miss_target": 0.1, "liftover": false, "target_build": "GRCh37", "min_lift": 0.95, "min_overlap": 0.0, "keep_ambiguous": false, "keep_multiallelic": false, "copy_genomes": false, "genotypes_cache": "/home/linux/urunner/system/test/cache", "only_bootstrap": false, "only_input": false, "only_compatible": false, "only_match": false, "only_projection": false, "only_score": false, "skip_ancestry": true, "outdir": "/home/linux/urunner/system/test/results", "publish_dir_mode": "copy", "email": null, "email_on_fail": null, "plaintext_email": false, "monochrome_logs": false, "hook_url": null, "help": false, "version": false, "config_profile_name": null, "config_profile_description": null, "custom_config_version": "master", "resume": true, "custom_config_base": "https://raw.githubusercontent.com/nf-core/configs/master", "config_profile_contact": null, "config_profile_url": null, "max_memory": "62.GB", "max_cpus": 8, "max_time": "240.h"

Fiwx commented 3 months ago

I reran pgsc_calc with --resume after a loop device error and got this:

linux@b3d36862d500:~/urunner/system/test$ nextflow run /home/linux/urunner/tools/pgsc_calc/main.nf -profile singularity --input /home/linux/urunner/system/test/u10000_file8267_yofb_uk_s_uk.23andme_output/samplesheet.csv --scorefile '/home/linux/urunner/system/test/custom_scores/*.txt.gz' --target_build GRCh37 --min_overlap 0.0 --pca_maf_target 0.0 --run_ancestry /home/linux/urunner/data/pgsc_1000G_v1.tar.zst -c /home/linux/urunner/references/custom.config --genotypes_cache /home/linux/urunner/system/test/cache --resume

 N E X T F L O W   ~  version 24.04.4
Launching `/home/linux/urunner/tools/pgsc_calc/main.nf` [determined_mayer] DSL2 - revision: a10e219a42
------------------------------------------------------
  pgscatalog/pgsc_calc v2.0.0-beta.3
------------------------------------------------------
Core Nextflow options
  runName        : determined_mayer
  containerEngine: singularity
  launchDir      : /home/linux/urunner/system/test
  workDir        : /home/linux/urunner/system/test/work  projectDir     : /home/linux/urunner/tools/pgsc_calc  userName       : linux
  profile        : singularity  configFiles    :
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------If you use pgscatalog/pgsc_calc for your analysis please cite:
* The Polygenic Score Catalog
  https://doi.org/10.1101/2024.05.29.24307783
  https://doi.org/10.1038/s41588-021-00783-5
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/pgscatalog/pgsc_calc/blob/main/CITATIONS.md
Reference database provided: skipping bootstrap
executor >  local (3)
[dd/84dcdb] PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)                                             [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM                                              -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR                                             -
[skipped  ] PGS…CALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, stored: 1 ✔
[skipped  ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)                                          [100%] 1 of 1, stored: 1 ✔
[cd/924b37] PGS…CALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [  0%] 0 of 1
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FILTER_VARIANTS                                               -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_REF                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_THINNED                                             -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:RELABEL_IDS                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_TARGET                                         -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_ORIENT                                                 -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PCA                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PROJECT                                               -
[db/237c48] PGS…G_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [  0%] 0 of 1
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE                                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:RELABEL_SCOREFILES                                                 -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:RELABEL_AFREQ                                                      -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:PLINK2_SCORE                                                       -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:SCORE_AGGREGATE                                                    -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:ANCESTRY_ANALYSIS                                                       -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT                                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:DUMPSOFTWAREVERSIONS                                                           -
ERROR ~ Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL)'

Caused by:
  Missing output file(s) `u10000file8267yofbuksuk_ALL_matched.txt.gz` expected by process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL)`

Command executed:

  pgscatalog-intersect --ref GRCh37_1000G_ALL.pvar.zst         --target GRCh37_u10000file8267yofbuksuk_ALL.pvar.zst         --chrom ALL         --maf_target 0.0         --geno_miss 0.1         --outdir .         -v

  n_matched=$(sed -n '3p' intersect_counts_ALL.txt)

  if [ $n_matched == "0" ]
  then
      echo "ERROR: No variants in intersection"
      exit 1
  else
      mv matched_variants.txt.gz u10000file8267yofbuksuk_ALL_matched.txt.gz
  fi

  cat <<-END_VERSIONS > versions.yml
  INTERSECT_VARIANTS:
      pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
  END_VERSIONS

Command exit status:
  0

Command output:
  (empty)

Command error:
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:12 INFO     Processed 6000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:18 INFO     Processed 6500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:24 INFO     Processed 7000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:30 INFO     Processed 7500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:36 INFO     Processed 8000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:42 INFO     Processed 8500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:48 INFO     Processed 9000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:54 INFO     Processed 9500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:40:00 INFO     Processed 10000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:40:06 INFO     Processed 10500000
[...]
TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:41:08 INFO     Processed 15500000 TARGET variants
executor >  local (3)
[dd/84dcdb] PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)                                             [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM                                              -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR                                             -
[skipped  ] PGS…CALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, stored: 1 ✔
[skipped  ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)                                          [100%] 1 of 1, stored: 1 ✔
[cd/924b37] PGS…CALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, failed: 1 ✘
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FILTER_VARIANTS                                               -
[-        ] 

[...]

  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:30 INFO     Processed 7500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:36 INFO     Processed 8000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:42 INFO     Processed 8500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:48 INFO     Processed 9000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:39:54 INFO     Processed 9500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:40:00 INFO     Processed 10000000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:40:06 INFO     Processed 10500000 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:40:12 INFO     Processed 11000000

[...]

TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:41:08 INFO     Processed 15500000 TARGET variants
executor >  local (3)
[dd/84dcdb] PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)                                             [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM                                              -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR                                             -
[skipped  ] PGS…CALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, stored: 1 ✔
[skipped  ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)                                          [100%] 1 of 1, stored: 1 ✔
[cd/924b37] PGS…CALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, failed: 1 ✘
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FILTER_VARIANTS                                               -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_REF                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_THINNED                                             -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:RELABEL_IDS                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_TARGET                                         -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_ORIENT                                                 -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PCA                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PROJECT                                               -
[db/237c48] PGS…G_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE                                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:APPLY_SCORE:RELABEL_SCOREFILES                                            

[...]

TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:41:08 INFO     Processed 15500000 TARGET variants
executor >  local (3)
[dd/84dcdb] PGSCATALOG_PGSCCALC:PGSCCALC:INPUT_CHECK:COMBINE_SCOREFILES (1)                                             [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELBIM                                              -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_RELABELPVAR                                             -
[skipped  ] PGS…CALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, stored: 1 ✔
[skipped  ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)                                          [100%] 1 of 1, stored: 1 ✔
[cd/924b37] PGS…CALC:ANCESTRY_PROJECT:INTERSECT_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1, failed: 1 ✘
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FILTER_VARIANTS                                               -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_REF                                            -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:INTERSECT_THINNED                                             -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:RELABEL_IDS                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_MAKEBED_TARGET                                         -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:PLINK2_ORIENT                                                 -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PCA                                                   -
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:FRAPOSA_PROJECT                                               -
[db/237c48] PGS…G_PGSCCALC:PGSCCALC:MATCH:MATCH_VARIANTS (u10000file8267yofbuksuk chromosome ALL) [100%] 1 of 1 ✔
[-        ] PGSCATALOG_PGSCCALC:PGSCCALC:MATCH:MATCH_COMBINE                                                            -

[...]

TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:43:40 INFO     Processed 27904794 TARGET variants
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:43:40 INFO     Outputting TARGET variants -> target_variants.txt.gz
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:45:05 INFO     Joining & outputting matched variants -> matched_variants.txt.gz
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     27904794/27904794 (100.00%) of TARGET variants matched the REFERENCE data
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     23599799/27904794 (84.57%) of matched variants are eligible for PCA
  pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     Outputting variant counts -> intersect_counts_ALL.txt

Work dir:
  /home/linux/urunner/system/test/work/cd/924b37caaa9eb2d2c92578efb38cfd

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

.command.out is empty.

.command.err ends in this:

[...]
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:43:40 INFO     Processed 27904794 TARGET variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:43:40 INFO     Outputting TARGET variants -> target_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:45:05 INFO     Joining & outputting matched variants -> matched_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     27904794/27904794 (100.00%) of TARGET variants matched the REFERENCE data
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     23599799/27904794 (84.57%) of matched variants are eligible for PCA
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     Outputting variant counts -> intersect_counts_ALL.txt

.command.log ends in this:

[...]

pgscatalog.match.cli.intersect_cli: 2024-08-25 17:43:40 INFO     Outputting TARGET variants -> target_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:45:05 INFO     Joining & outputting matched variants -> matched_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     27904794/27904794 (100.00%) of TARGET variants matched the REFERENCE data
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     23599799/27904794 (84.57%) of matched variants are eligible for PCA
pgscatalog.match.cli.intersect_cli: 2024-08-25 17:51:18 INFO     Outputting variant counts -> intersect_counts_ALL.txt
cp: '.command.out' and '/home/linux/urunner/system/test/work/cd/924b37caaa9eb2d2c92578efb38cfd/.command.out' are the same file
cp: '.command.err' and '/home/linux/urunner/system/test/work/cd/924b37caaa9eb2d2c92578efb38cfd/.command.err' are the same file
cp: '.command.trace' and '/home/linux/urunner/system/test/work/cd/924b37caaa9eb2d2c92578efb38cfd/.command.trace' are thesame file
mkdir: cannot create directory '/home/linux/urunner/system/test/cache/ancestry/intersected': Permission denied
mv: cannot move 'intersect_counts_ALL.txt' to '/home/linux/urunner/system/test/cache/ancestry/intersected/.': No such file or directory
mkdir: cannot create directory '/home/linux/urunner/system/test/cache/ancestry/intersected': Permission denied
mv: cannot move 'user10000file8266yearofbirthunknownsexunknown_ALL_matched.txt.gz' to '/home/linux/urunner/system/test/cache/ancestry/intersected/.': No such file or directory
mkdir: cannot create directory '/home/linux/urunner/system/test/cache/ancestry/intersected': Permission denied
mv: cannot move 'versions.yml' to '/home/linux/urunner/system/test/cache/ancestry/intersected/.': No such file or directory

.command.sh:
#!/bin/bash -euo pipefail
pgscatalog-intersect --ref GRCh37_1000G_ALL.pvar.zst         --target GRCh37_user10000file8266yearofbirthunknownsexunknown_ALL.pvar.zst         --chrom ALL         --maf_target 0.0         --geno_miss 0.1         --outdir .         -v

n_matched=$(sed -n '3p' intersect_counts_ALL.txt)

if [ $n_matched == "0" ]
then
    echo "ERROR: No variants in intersection"
    exit 1
else
    mv matched_variants.txt.gz user10000file8266yearofbirthunknownsexunknown_ALL_matched.txt.gz
fi

cat <<-END_VERSIONS > versions.yml
INTERSECT_VARIANTS:
    pgscatalog.match: $(echo $(python -c 'import pgscatalog.match; print(pgscatalog.match.__version__)'))
END_VERSIONS

.command.trace:

nextflow.trace/v2
realtime=1854040
%cpu=1018
cpu_model=Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
rchar=7668009450
wchar=6997428096
syscr=704235
syscw=698891
read_bytes=555186176
write_bytes=6997999616
%mem=5
vmem=608252
rss=316464
peak_vmem=608252
peak_rss=316464
vol_ctxt=62315
inv_ctxt=2864
Fiwx commented 3 months ago

I ran the exact command that was in .command.sh, and it succeeded without an error:

linux@b3d36862d500:~/urunner/system/test/work/cd/924b37caaa9eb2d2c92578efb38cfd$ pgscatalog-intersect --ref GRCh37_1000G_ALL.pvar.zst         --target GRCh37_user10000file8266yearofbirthunknownsexunknown_ALL.pvar.zst         --chrom ALL         --maf_target 0.0         --geno_miss 0.1         --outdir .         -v
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:22:56 DEBUG    Verbose logging enabled
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:22:56 INFO     Reading REFERENCE variants: GRCh37_1000G_ALL.pvar.zst
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:23:01 INFO     Processed 500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:23:05 INFO     Processed 1000000 REFERENCE variants
[...]
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:35:46 INFO     Processed 84500000 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:35:49 INFO     Processed 84805772 REFERENCE variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:35:49 INFO     Outputting REFERNCE variants -> reference_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:40:06 INFO     Reading TARGET variants: GRCh37_user10000file8266yearofbirthunknownsexunknown_ALL.pvar.zst
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:40:12 INFO     Processed 500000 TARGET variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:40:19 INFO     Processed 1000000 TARGET variants
[...]
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:45:35 INFO     Processed 27904794 TARGET variants
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:45:35 INFO     Outputting TARGET variants -> target_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:46:52 INFO     Joining & outputting matched variants -> matched_variants.txt.gz
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:52:27 INFO     27904794/27904794 (100.00%) of TARGET variants matched the REFERENCE data
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:52:27 INFO     23599799/27904794 (84.57%) of matched variants are eligible for PCA
pgscatalog.match.cli.intersect_cli: 2024-08-25 18:52:27 INFO     Outputting variant counts -> intersect_counts_ALL.txt

The original loop devices error happens with and without this recommendation: https://github.com/PGScatalog/pgsc_calc/issues/361#issuecomment-2296923470, and on both 64 and 32 GB of RAM, and also with either 65 or 93 scorefiles.

smlmbrt commented 3 months ago

Hmmm, we might have to investigate this further. It could also be some interaction with how singularity was initially set up/administered. You could also try conda to see if that works?

Fiwx commented 3 months ago

I will look into conda. The loop devices themselves look normal. Perhaps it is some setup issue that occurs under certain conditions, since I previously used the same setup without seeing this error.

Fiwx commented 2 months ago

I can still replicate the error but I don't think it's the root cause of the pipeline failing to run:

Sep-12 19:41:35.981 [main] DEBUG nextflow.Session - Failed to invoke observer completion handler: nextflow.prov.ProvObserver@4a9e4be7
java.lang.NullPointerException: null
        at nextflow.prov.BcoRenderer$_render_closure3.doCall(BcoRenderer.groovy:93)
        at nextflow.prov.BcoRenderer$_render_closure3.call(BcoRenderer.groovy)
        ...
        at nextflow.cli.Launcher.main(Launcher.groovy:657)
nebfield commented 2 months ago

Does the problem still happen if you set:

https://github.com/PGScatalog/pgsc_calc/blob/96fbb2346978c12917d35dc68520a86a9f7b7cc0/nextflow.config#L306-L313

to enabled = false?

Fiwx commented 2 months ago

I'll try that next, thanks.

The loop device might be a different issue since I got this error without the loop device message:

PGSCATALOG_PGSCCALC:PGSCCALC:REPORT:SCORE_REPORT (u83); status: COMPLETED; exit: 0; error: -; workDir: /home/group/user/runner/current/u83/work/9a/05173663b3b5fa61bc3a1aa7ef1178]
Sep-21 13:00:19.838 [main] DEBUG nextflow.Session - Session await > all processes finished
Sep-21 13:00:19.929 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Sep-21 13:00:19.929 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sep-21 13:00:19.934 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Sep-21 13:00:19.936 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Sep-21 13:00:19.964 [main] INFO  nextflow.Nextflow - -[pgscatalog/pgsc_calc] Pipeline completed successfully-
Sep-21 13:00:19.973 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=27; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=11h 23m 12s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=8; peakMemory=56 GB; ]
Sep-21 13:00:19.973 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Sep-21 13:00:19.975 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Sep-21 13:00:20.858 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Sep-21 13:00:21.143 [main] DEBUG nextflow.Session - Failed to invoke observer completion handler: nextflow.prov.ProvObserver@40d2a644
java.lang.NullPointerException: null
    at nextflow.prov.BcoRenderer$_render_closure3.doCall(BcoRenderer.groovy:93)
    at nextflow.prov.BcoRenderer$_render_closure3.call(BcoRenderer.groovy)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3661)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3646)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3692)
    at nextflow.prov.BcoRenderer.render(BcoRenderer.groovy:91)
    at nextflow.prov.ProvObserver$_onFlowComplete_closure4.doCall(ProvObserver.groovy:111)
    at nextflow.prov.ProvObserver$_onFlowComplete_closure4.call(ProvObserver.groovy)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2394)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2379)
    at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2420)
    at nextflow.prov.ProvObserver.onFlowComplete(ProvObserver.groovy:110)
    at nextflow.Session.notifyFlowComplete(Session.groovy:1115)
    at nextflow.Session.shutdown0(Session.groovy:738)
    at nextflow.Session.destroy(Session.groovy:683)
    at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:260)
    at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:146)
    at nextflow.cli.CmdRun.run(CmdRun.groovy:372)
    at nextflow.cli.Launcher.run(Launcher.groovy:503)
    at nextflow.cli.Launcher.main(Launcher.groovy:657)

command.sh:

#!/usr/bin/env python
import yaml
import platform
from textwrap import dedent
def _make_versions_html(versions):
    html = [
        dedent(
            '''\
            <style>
            #nf-core-versions tbody:nth-child(even) {
                background-color: #f2f2f2;
            }
            </style>
            <table class="table" style="width:100%" id="nf-core-versions">
                <thead>
                    <tr>
                        <th> Process Name </th>
                        <th> Software </th>
                        <th> Version  </th>
                    </tr>
                </thead>
            '''
        )
    ]
    for process, tmp_versions in sorted(versions.items()):
        html.append("<tbody>")
        for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
            html.append(
                dedent(
                    f'''\
                    <tr>
                        <td><samp>{process if (i == 0) else ''}</samp></td>
                        <td><samp>{tool}</samp></td>
                        <td><samp>{version}</samp></td>
                    </tr>
                    '''
                )
            )
        html.append("</tbody>")
    html.append("</table>")
    return "\n".join(html)
module_versions = {}
module_versions["DUMPSOFTWAREVERSIONS"] = {
    'python': platform.python_version(),
    'yaml': yaml.__version__
}
with open("collated_versions.yml") as f:
    workflow_versions = yaml.load(f, Loader=yaml.BaseLoader) | module_versions
workflow_versions["Workflow"] = {
    "Nextflow": "24.04.4",
    "pgscatalog/pgsc_calc": "2.0.0-beta.3"
}
versions_mqc = {
    'id': 'software_versions',
    'section_name': 'pgscatalog/pgsc_calc Software Versions',
    'section_href': 'https://github.com/pgscatalog/pgsc_calc',
    'plot_type': 'html',
    'description': 'are collected at run time from the software output.',
    'data': _make_versions_html(workflow_versions)
}
with open("software_versions.yml", 'w') as f:
    yaml.dump(workflow_versions, f, default_flow_style=False)
with open("software_versions_mqc.yml", 'w') as f:
    yaml.dump(versions_mqc, f, default_flow_style=False)
with open('versions.yml', 'w') as f:
    yaml.dump(module_versions, f, default_flow_style=False)

This part isn't a critical error since all of the data and results are still present.

Fiwx commented 2 months ago

This time, running the same pipeline, I got this error without the BcoRenderer issue. I would usually be able to provide more informative information but it seems like I can't fully reproduce the exact error each time despite having a very similar or exactly the same setup, for some reason.

…
PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1); status: COMPLETED; exit: 255; error: -; workDir: /home/user/c/crundir/worker/u10_file8297_yofu10_unknown_sa_unknown.2/work/af/34081c90497a21ca2eadb049320e6f]
Sep-22 11:44:27.491 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Sep-22 11:44:27.501 [TaskFinalizer-1] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1); work-dir=/home/user/c/crundir/worker/u10_file8297_yofu10_unknown_sa_unknown.2/work/af/34081c90497a21ca2eadb049320e6f
  error [nextflow.exception.ProcessFailedException]: Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)` terminated with an error exit status (255)
Sep-22 11:44:27.540 [TaskFinalizer-1] ERROR nextflow.processor.TaskProcessor - Error executing process > 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)'

Caused by:
  Process `PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)` terminated with an error exit status (255)

Command executed:

  tar -xf pgsc_1000G_v1.tar.zst --wildcards "GRCh37*" meta.txt 2> /dev/null

  DB_VERSION=$(cat meta.txt)

  if [ "$DB_VERSION" != "v0.1" ]; then
    echo "Old reference database version detected, please redownload the latest version and try again"
    echo "See https://pgsc-calc.readthedocs.io/en/latest/how-to/ancestry.html"
    exit 1
  else
    echo "Database version good"
  fi

  cat <<-END_VERSIONS > versions.yml
  EXTRACT_DATABASE:
      zstd: $(zstd -V | grep -Eo 'v[0-9]\.[0-9]\.[0-9]+' )
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  FATAL:   container creation failed: mount /proc/self/fd/4->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/4: failed to find loop device: could not attach image file to loop device: no loop devices available
  cp: '.command.out' and '.command.out' are the same file
  cp: '.command.err' and '.command.err' are the same file
  cp: cannot stat '.command.trace': No such file or directory

Work dir:
  /home/user/c/crundir/worker/u10_file8297_yofu10_unknown_sa_unknown.2/work/af/34081c90497a21ca2eadb049320e6f

.command.run:

#!/bin/bash
### ---
### name: 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)'
[...]
    set -u
    [[ $NXF_SCRATCH ]] && cd $NXF_SCRATCH
    export NXF_TASK_WORKDIR="$PWD"
    nxf_stage

    set +e
    (set -o pipefail; (nxf_launch | tee .command.out) 3>&1 1>&2 2>&3 | tee .command.err) &
    pid=$!
    wait $pid || nxf_main_ret=$?
    nxf_unstage
}

$NXF_ENTRY

.command.sh:

#!/bin/bash -euo pipefail
tar -xf pgsc_1000G_v1.tar.zst --wildcards "GRCh37*" meta.txt 2> /dev/null

DB_VERSION=$(cat meta.txt)

if [ "$DB_VERSION" != "v0.1" ]; then
  echo "Old reference database version detected, please redownload the latest version and try again"
  echo "See https://pgsc-calc.readthedocs.io/en/latest/how-to/ancestry.html"
  exit 1
else
  echo "Database version good"
fi

cat <<-END_VERSIONS > versions.yml
EXTRACT_DATABASE:
    zstd: $(zstd -V | grep -Eo 'v[0-9]\.[0-9]\.[0-9]+' )
END_VERSIONS

In the work directory, there is this file: pgsc_1000G_v1.tar.zst. (Also, I'm not able to get Conda working in the environment. I'm running the newest release of pgsc_calc with Singularity. I run 1 VCF file at a time. Between different VCFs, this error does not always happen, but it happened on a large percentage of runs/files (four out of five runs). However, I'm not sure that the files are the issue since I remember that some of the files completed without this error on other runs yesterday.)

Fiwx commented 2 months ago

I switched to enabled = false, but it still crashes almost always with the loop devices issue.

The last running process on four of the times I got the loop device error was: pgscatalog-comb, pgscatalog-comb, pgscatalog-rela, plink2

For the above 4 cases, the nextflow logs show if failed on: PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE, for pgscatalog-comb, pgscatalog-comb

PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF for pgscatalog-rela

PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE for plink2

Here are some excerpts from the work directory command.run files:

### name: 'PGSCATALOG_PGSCCALC:PGSCCALC:MAKE_COMPATIBLE:PLINK2_VCF (u8ukfile chromosome ALL)'
### container: '/home/user/containers/ghcr.io-pgscatalog-plink2-2.00a5.10-singularity.img'

... 

## name: 'PGSCATALOG_PGSCCALC:PGSCCALC:ANCESTRY_PROJECT:EXTRACT_DATABASE (1)'
### container: '/home/user/containers/ghcr.io-pgscatalog-zstd-2-beta-singularity.img'