the pipeline cannot find "fastq.gz"

I ran "nextflow run" according to the instructions "Run the workflow for mouse data with conda," but it seems that the fastq.gz files are not being recognized correctly. Even though I specified the directory where the fastq.gz files are stored using "--fqdir," I'm getting an error saying "No fastq file matching '.fastq.gz' in ...". Following the comment "Check script 'main.nf' at line: 119," I checked it, but everything seems fine. Could you please advise on how to address this issue?

Here is the command I used for running nextflow (version 23.04.1): nextflow run /hoge/ribomethseq-nf-main/ -dsl2 -profile mouse,conda --fqdir /hoge/ribomethseq-nf-main/fastq_folder --outdir /hoge/ribomethseq-nf-main/outputs

Note: The latest version of nextflow, 23.04.1, was used, and it seems that it doesn't run properly without the "-dsl2" option.

Thank you for reporting this issue :slightly_smiling_face: !

There is no problem with your command. I have also tested to run ribomethseq-nf with Nextflow 23.04.1 and have indeed the same issue.

This is caused by the newer versions of Nextflow, as they have dropped support for DSL1. This also explains why Nextflow is now asking for the "-dsl2" option.

We will soon migrate ribomethseq-nf from DSL1 to DSL2. While we work on the migration, you can run the current version of ribomethseq-nf with Nextflow 22.10.4.

Thank you for a good suggestion. I have changed the Nextflow version from 23.04.1 to 22.10.4, and this change resolved the previous error I encountered. However, I'm now facing another error message:

class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')

I attempted to resolve this issue by changing the Java versions to 20 and 17, but it did not work. It's possible that this error is occurring due to an incorrect directory where the Java files are located on my computer, but I'm unsure how to address it. Could you please help me troubleshoot this problem?

I think this error happens inside Nextflow than in Java itself. It is indeed very unlikely that changing Java's version will resolve this issue.

class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')

Unfortunately, this error message gives no detail on where Nextflow has really failed.

You can find more informations about the error in the .nextflow.log file. This (hidden) file is located at /hoge/ribomethseq-nf-main/.nextflow.log.

Can you copy and paste the part where the error happens ?

For example, this is the kind of error/warning message you can find in this file :

juin-04 22:08:15.437 [main] WARN  nextflow.trace.ReportObserver - Failed to render execution report -- see the log file for details
nextflow.exception.AbortOperationException: Report file already exists: /home/theo/ribomethseq-nf/results/ribomethseq-nf.report/exec-report.html -- enable the 'report.overwrite' option in your>
        at nextflow.trace.TraceHelper.newFileWriter(TraceHelper.groovy:66)
        at nextflow.trace.ReportObserver.renderHtml(ReportObserver.groovy:282)
        at nextflow.trace.ReportObserver.onFlowComplete(ReportObserver.groovy:139)
        at nextflow.Session.notifyFlowComplete(Session.groovy:1074)
        at nextflow.Session.shutdown0(Session.groovy:702)
        at nextflow.Session.destroy(Session.groovy:644)
        at nextflow.script.ScriptRunner.shutdown(ScriptRunner.groovy:239)
        at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:134)
        at nextflow.cli.CmdRun.run(CmdRun.groovy:368)
        at nextflow.cli.Launcher.run(Launcher.groovy:494)
        at nextflow.cli.Launcher.main(Launcher.groovy:653)
Caused by: java.nio.file.FileAlreadyExistsException: /home/theo/ribomethseq-nf/results/ribomethseq-nf.report/exec-report.html
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
        at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
        at java.base/java.nio.file.Files.newOutputStream(Files.java:220)
        at java.base/java.nio.file.Files.newBufferedWriter(Files.java:2920)
        at nextflow.trace.TraceHelper.newFileWriter(TraceHelper.groovy:63)
        ... 10 common frames omitted

If you can find the complete error message that you have in your own .nextflow.log file, that would help me a lot to understand where is really the issue.

I will also ask for some information and give some solution to test :

If you are trying to run the pipeline in a cluster environment and are unsure on how to run Nextflow and Conda on this cluster, I would first recommend you to contact the system administrators or their technical support for help.

What operating system are you using ?
Is Conda installed on the machine where you run the pipeline ?
Are you running the pipeline inside a cluster ?
If you run the pipeline inside a cluster, what is the cluster's job scheduler ?

By default, the config file /hoge/ribomethseq-nf-main/nf-config/exec.config specifies Slurm as the job scheduler : params.scheduler = 'slurm'.

If this is not your case, you will have to change 'slurm' to the corresponding job scheduler of your cluster. For example, if your cluster uses Sun Grid Engine instead of Slurm, the first line in exec.config should be :

params.scheduler = 'sge'

You can check what Job scheduler / process executor Nextflow supports here : https://www.nextflow.io/docs/latest/executor.html

If you are running the pipeline on a local computer and there is no job scheduler, you can replace 'slurm' with 'local' : params.scheduler = 'local'

On a side note, I have migrated most of the pipeline to DSL2, so it will be soon possible to run it with Nextflow 23.04.1 :grin:. I'm not sure this will fix your second issue though... :thinking:

I have extracted the relevant portion of the error message. I am currently running the pipeline on my local computer, so I have modified the job scheduler to 'local'. I am using macOS (version 10.15.7) and have Conda installed on my local machine. I hope the cause of the error can be found from the logs. Thank you in advance for your assistance.

6月-05 20:38:02.021 [main] INFO nextflow.Nextflow - 6月-05 20:38:02.065 [PathVisitor-1] ERROR nextflow.Channel - class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap') java.lang.ClassCastException: class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap') at com.sun.proxy.$Proxy27.getFileSystem(Unknown Source) at nextflow.file.PathVisitor.applyGlobPattern0(PathVisitor.groovy:115) at nextflow.file.PathVisitor.apply(PathVisitor.groovy:69) at nextflow.file.PathVisitor$_applyAsync_closure1.doCall(PathVisitor.groovy:78) at nextflow.file.PathVisitor$_applyAsync_closure1.call(PathVisitor.groovy) at groovy.lang.Closure.run(Closure.java:493) at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:783) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 6月-05 20:38:02.075 [PathVisitor-1] DEBUG nextflow.Session - Session aborted -- Cause: class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')

Thank you ! That's exactly what I wanted ! Your error seems to happen when Nextflow tries to access the fastq.gz files in fastqc_folder. That's weird... :raised_eyebrow:. I would like to see the full .nextflow.log just to be sure I am not missing something.

I have never tried ribomethseq-nf on MacOS. But that should not be an issue. (I will test on a Mac very soon, just to be sure)

Could you give me the result of the following command ? java --version

Output example :

openjdk 10.0.2 2018-07-17
OpenJDK Runtime Environment Zulu10.3+5 (build 10.0.2+13)
OpenJDK 64-Bit Server VM Zulu10.3+5 (build 10.0.2+13, mixed mode)

You can also try to install Nextflow inside a new Conda environment. Conda should install java (openjdk) alongside of it : conda create -n nextflowEnv nextflow=22.10.6

Note 1 : I have set Nextflow's version to 22.10.6 Note 2 : I have named the environment nextflowEnv. You can change this name.

And then run the pipeline by using the newly installed Nextflow inside this environment :

conda activate nextflowEnv

nextflow run /hoge/ribomethseq-nf-main/ -profile mouse,conda --fqdir /hoge/ribomethseq-nf-main/fastq_folder --outdir /hoge/ribomethseq-nf-main/outputs

Thank you for your suggestion! I generated a new conda environment as you suggested. But unfortunately, I got the same error.

Here is the java version. openjdk 11.0.13 2021-10-19 OpenJDK Runtime Environment JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21) OpenJDK 64-Bit Server VM JBR-11.0.13.7-1751.21-jcef (build 11.0.13+7-b1751.21, mixed mode)

How should I send you the full .nextflow.log?

Thank you !

My message is a little bit long, so I have divided it in three parts :

Nextflow log

You can either copy and paste or attach your log file in your message. The log should not be too long, as the pipeline fails at the beginning.

Solution 1 : copy and paste your log directly in your message.

You can paste your log directly in your message.

Github also supports markdown formatting, so you can put your log inside a block of code using three backticks (```). Their help article.

You can write the following in your message : \``` paste your log here \``` This will give the following result : ``` paste your log here ```

Solution 2 : attach the file directly to your message If you prefer to attach a file than copy and paste, you can follow Github's help article.

Input fastq.gz

if you type find /hoge/ribomethseq-nf-main/fastq_folder in your terminal, it will return a list of all the files and directories (if any) inside fastq_folder.

Can you give me the output of this command ?

Example :

> find ./tests/fastq/

./tests/fastq/
./tests/fastq/sample2_R1_001.fastq.gz
./tests/fastq/sample3_R1_001.fastq.gz
./tests/fastq/sample1_R1_001.fastq.gz

Running the pipeline with our dataset

Can you try to run the pipeline with our test dataset ?

The test dataset is located in /hoge/ribomethseq-nf-main/tests/fastq/

the command should be nextflow run /hoge/ribomethseq-nf-main/ -profile mouse,conda --fqdir /hoge/ribomethseq-nf-main/tests/fastq --outdir /hoge/ribomethseq-nf-main/outputs-test

I have renamed the output folder in the command to outputs-test.

I've just answered your comment point by point. Hopefully, this will give you some clues.

1. Nextflow log

Here is the full log file, which is a bit long.

6月-10 09:32:04.742 [main] DEBUG nextflow.cli.Launcher - $> nextflow run /Users//Desktop/Ribometh/ribomethseq-nf-main --logdir Outdir -profile mouse,conda --fqdir /Users//Desktop/Ribometh/ribomethseq-nf-main/tests/fastq --outdir /Users//Desktop/Ribometh/ribomethseq-nf-main/outputs-test
6月-10 09:32:04.857 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 22.10.4
6月-10 09:32:04.888 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/NAME/.nextflow/plugins; core-plugins: nf-amazon@1.11.2,nf-azure@0.14.2,nf-codecommit@0.1.2,nf-console@1.0.4,nf-ga4gh@1.0.4,nf-google@1.4.5,nf-tower@1.5.6,nf-wave@0.5.2
6月-10 09:32:04.904 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
6月-10 09:32:04.905 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
6月-10 09:32:04.909 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
6月-10 09:32:04.923 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
6月-10 09:32:05.714 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/nextflow.config
6月-10 09:32:05.715 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/nextflow.config
6月-10 09:32:05.728 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `mouse,conda`
6月-10 09:32:05.908 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [mouse, conda, singularity, human, docker]
6月-10 09:32:05.947 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=1 by probing script field
6月-10 09:32:05.975 [main] INFO  nextflow.cli.CmdRun - Launching `/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/main.nf` [tender_mcnulty] DSL1 - revision: 2c33db8fe2
6月-10 09:32:05.976 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
6月-10 09:32:05.976 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
6月-10 09:32:05.987 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /Users/NAME/.nextflow/secrets/store.json
6月-10 09:32:05.991 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@60990e5c] - activable => nextflow.secret.LocalSecretsProvider@60990e5c
6月-10 09:32:06.073 [main] DEBUG nextflow.Session - Session UUID: 6e77a512-d127-4b81-b2ea-c5d94f0ec1f4
6月-10 09:32:06.073 [main] DEBUG nextflow.Session - Run : tender_mcnulty
6月-10 09:32:06.073 [main] DEBUG nextflow.Session - Executor pool size: 8
6月-10 09:32:06.088 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
6月-10 09:32:06.369 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 22.10.4 build 5836
  Created: 09-12-2022 09:58 UTC (18:58 JDT)
  System: Mac OS X 10.15.7
  Runtime: Groovy 3.0.13 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21
  Encoding: UTF-8 (UTF-8)
  Process: 58468@-MacBook-Pro.local [127.0.0.1]
  CPUs: 8 - Mem: 32 GB (12.6 GB) - Swap: 12 GB (1.2 GB)
6月-10 09:32:06.385 [main] DEBUG nextflow.Session - Work-dir: /Users//Desktop/Ribometh/ribomethseq-nf-main/work [Mac OS X]
6月-10 09:32:06.386 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users//Desktop/Ribometh/ribomethseq-nf-main/bin
6月-10 09:32:06.403 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
6月-10 09:32:06.415 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
6月-10 09:32:06.491 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
6月-10 09:32:06.508 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
6月-10 09:32:06.617 [main] DEBUG nextflow.Session - Session start
6月-10 09:32:06.624 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/Outdir/ribomethseq-nf.report/exec-trace.txt
6月-10 09:32:07.468 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
6月-10 09:32:07.481 [main] INFO  nextflow.Nextflow - 
6月-10 09:32:07.481 [main] INFO  nextflow.Nextflow - --------------------------------------------------------------------
6月-10 09:32:07.482 [main] INFO  nextflow.Nextflow -       ribomethseq-nf: RiboMethSeq data processing       
6月-10 09:32:07.482 [main] INFO  nextflow.Nextflow - --------------------------------------------------------------------
6月-10 09:32:07.482 [main] INFO  nextflow.Nextflow - Copyright (C) CRCL
6月-10 09:32:07.482 [main] INFO  nextflow.Nextflow - This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE
6月-10 09:32:07.482 [main] INFO  nextflow.Nextflow - This is free software, and you are welcome to redistribute it
6月-10 09:32:07.483 [main] INFO  nextflow.Nextflow - under certain conditions; see LICENSE for details.
6月-10 09:32:07.483 [main] INFO  nextflow.Nextflow - --------------------------------------------------------------------
6月-10 09:32:07.483 [main] INFO  nextflow.Nextflow - 
6月-10 09:32:07.496 [main] INFO  nextflow.Nextflow - ----------------------- Workflow properties ------------------------
6月-10 09:32:07.497 [main] INFO  nextflow.Nextflow - Workflow project directory       : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main
6月-10 09:32:07.497 [main] INFO  nextflow.Nextflow - Workflow version                 : 1.0
6月-10 09:32:07.498 [main] INFO  nextflow.Nextflow - Workflow launched from           : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main
6月-10 09:32:07.498 [main] INFO  nextflow.Nextflow - Workflow workDir                 : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/work
6月-10 09:32:07.498 [main] INFO  nextflow.Nextflow - Workflow config file(s)          : [/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/nextflow.config]
6月-10 09:32:07.499 [main] INFO  nextflow.Nextflow - Workflow profile                 : mouse,conda
6月-10 09:32:07.499 [main] INFO  nextflow.Nextflow - Executed command line            : nextflow run /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main --logdir Outdir -profile mouse,conda --fqdir /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/tests/fastq --outdir /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/outputs-test
6月-10 09:32:07.499 [main] INFO  nextflow.Nextflow - 
6月-10 09:32:07.499 [main] INFO  nextflow.Nextflow - ------------------------ Execution parameters ----------------------
6月-10 09:32:07.499 [main] INFO  nextflow.Nextflow - Fastq input directory            : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/tests/fastq/
6月-10 09:32:07.500 [main] INFO  nextflow.Nextflow - Fastq selection pattern          : .fastq.gz
6月-10 09:32:07.500 [main] INFO  nextflow.Nextflow - Trimmomatic ILLUMINACLIP         : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/data/adapters/TruSeq3-SE.fa
6月-10 09:32:07.500 [main] INFO  nextflow.Nextflow - Trimmomatic LEADING              : 30
6月-10 09:32:07.500 [main] INFO  nextflow.Nextflow - Trimmomatic TRAILING             : 30
6月-10 09:32:07.500 [main] INFO  nextflow.Nextflow - Trimmomatic SLIDINGWINDOW        : 4:15
6月-10 09:32:07.501 [main] INFO  nextflow.Nextflow - Trimmomatic AVGQUAL              : 30
6月-10 09:32:07.501 [main] INFO  nextflow.Nextflow - Trimmomatic MINLEN               : 8
6月-10 09:32:07.501 [main] INFO  nextflow.Nextflow - Bowtie options                   : --sensitive -L 17
6月-10 09:32:07.501 [main] INFO  nextflow.Nextflow - Bowtie index                     : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/data/bowtie/mouse/mouse_index
6月-10 09:32:07.501 [main] INFO  nextflow.Nextflow - samtools options                 : --no-PG -h -u -d 'NM' -e '![XS]'
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - split count files by rna         : false
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - Threads for bowtie               : 7
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - Threads for fastqc               : 2
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - Threads for trimmomatic          : 3
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - Threads for samtools             : 4
6月-10 09:32:07.502 [main] INFO  nextflow.Nextflow - Export 3'end read count          : false
6月-10 09:32:07.503 [main] INFO  nextflow.Nextflow - Log directory                    : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/Outdir
6月-10 09:32:07.503 [main] INFO  nextflow.Nextflow - Output directory                 : /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/outputs-test
6月-10 09:32:07.503 [main] INFO  nextflow.Nextflow - Max parallel jobs                : 20
6月-10 09:32:07.503 [main] INFO  nextflow.Nextflow - 
6月-10 09:32:07.552 [PathVisitor-1] ERROR nextflow.Channel - class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')
java.lang.ClassCastException: class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')
    at com.sun.proxy.$Proxy27.getFileSystem(Unknown Source)
    at nextflow.file.PathVisitor.applyGlobPattern0(PathVisitor.groovy:115)
    at nextflow.file.PathVisitor.apply(PathVisitor.groovy:69)
    at nextflow.file.PathVisitor$_applyAsync_closure1.doCall(PathVisitor.groovy:78)
    at nextflow.file.PathVisitor$_applyAsync_closure1.call(PathVisitor.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:783)
    at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
6月-10 09:32:07.562 [PathVisitor-1] DEBUG nextflow.Session - Session aborted -- Cause: class java.lang.String cannot be cast to class java.nio.file.FileSystem (java.lang.String and java.nio.file.FileSystem are in module java.base of loader 'bootstrap')
6月-10 09:32:07.697 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc` matches process fastqc
6月-10 09:32:07.698 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process fastqc
6月-10 09:32:07.700 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.700 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.708 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
6月-10 09:32:07.715 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=8; memory=32 GB; capacity=20; pollInterval=100ms; dumpInterval=5m
6月-10 09:32:07.795 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:trim` matches process trim
6月-10 09:32:07.796 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process trim
6月-10 09:32:07.796 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.796 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.806 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:bowtie2` matches process bowtie2
6月-10 09:32:07.806 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process bowtie2
6月-10 09:32:07.807 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.807 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.813 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:filter` matches process filter
6月-10 09:32:07.813 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process filter
6月-10 09:32:07.814 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.814 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.828 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:multiqc` matches process multiqc
6月-10 09:32:07.828 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process multiqc
6月-10 09:32:07.829 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.829 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.839 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:counts` matches process counts
6月-10 09:32:07.839 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:fastqc|trim|bowtie2|filter|multiqc|counts` matches process counts
6月-10 09:32:07.840 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.840 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.848 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:split` matches process split
6月-10 09:32:07.848 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:split|report` matches process split
6月-10 09:32:07.848 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.848 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.857 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:report` matches process report
6月-10 09:32:07.857 [main] DEBUG nextflow.script.ProcessConfig - Config settings `with:split|report` matches process report
6月-10 09:32:07.858 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
6月-10 09:32:07.858 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
6月-10 09:32:07.863 [main] DEBUG nextflow.Session - Workflow process s [dsl1]: filter, bowtie2, split, trim, fastqc, counts, report, multiqc
6月-10 09:32:07.864 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
6月-10 09:32:07.864 [main] DEBUG nextflow.Session - Session await
6月-10 09:32:07.864 [main] DEBUG nextflow.Session - Session await > all processes finished
6月-10 09:32:07.864 [main] DEBUG nextflow.Session - Session await > all barriers passed
6月-10 09:32:07.875 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
6月-10 09:32:07.876 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
6月-10 09:32:07.877 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
6月-10 09:32:07.884 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
  []
6月-10 09:32:08.446 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
6月-10 09:32:08.481 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

2. Input fastq.gz

Here is my fastq files. I believe that the directory of these files are correct.

find /Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/880-3_S3_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/.DS_Store
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/879-2_S2_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/887-10_S10_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/888-11_S11_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/883-6_S6_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/881-4_S4_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/889-12_S12_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/886-9_S9_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/884-7_S7_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/885-8_S8_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/882-5_S5_R1.fastq.gz
/Users/NAME/Desktop/Ribometh/ribomethseq-nf-main/fastq/878-1_S1_R1.fastq.gz

3. Running the pipeline with our dataset

I ran a pipeline with test fastq files in the same way. But I somehow got the same error as attached above.

Hi,

For information, I did not manage to reproduce the bug. See execution trace below.

It rather looks like an issue with Nextflow/Groovy. How did you downgrade from nextflow 23.04.1 to 22.10.4 ?

Perhaps you should totally remove the .nextflow directory in your home directory before installing 22.10.4 from a clean environment.

I run locally on a Mac OS X Ventura 13.3.1.

$ nextflow -version

      N E X T F L O W
      version 22.10.6 build 5843
      created 23-01-2023 23:20 UTC (24-01-2023 00:20 CEST)
      cite doi:10.1038/nbt.3820
      http://nextflow.io

$ java --version
java 14.0.1 2020-04-14
Java(TM) SE Runtime Environment (build 14.0.1+7)
Java HotSpot(TM) 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

$ cd tests
$ make test-conda SCHEDULER=local

Launching `../main.nf` [infallible_baekeland] DSL1 - revision: 35ea575212

--------------------------------------------------------------------
      ribomethseq-nf: RiboMethSeq data processing
--------------------------------------------------------------------
Copyright (C) CRCL
This program comes with ABSOLUTELY NO WARRANTY; for details see LICENSE
This is free software, and you are welcome to redistribute it
under certain conditions; see LICENSE for details.
--------------------------------------------------------------------

----------------------- Workflow properties ------------------------
Workflow project directory       : /Users/ferrari/Work/Sandbox/ribomethseq-nf
Workflow version                 : 1.0
Workflow launched from           : /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests
Workflow workDir                 : /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/work
Workflow config file(s)          : [/Users/ferrari/Work/Sandbox/ribomethseq-nf/nextflow.config]
Workflow profile                 : human,conda
Executed command line            : nextflow -bg run .. -profile human,conda --scheduler local --qsize 6 --outdir ribomethseq --logdir . --fqdir fastq --fastq_pattern .fastq.gz

------------------------ Execution parameters ----------------------
Fastq input directory            : /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/fastq/
Fastq selection pattern          : .fastq.gz
Trimmomatic ILLUMINACLIP         : /Users/ferrari/Work/Sandbox/ribomethseq-nf/data/adapters/TruSeq3-SE.fa
Trimmomatic LEADING              : 30
Trimmomatic TRAILING             : 30
Trimmomatic SLIDINGWINDOW        : 4:15
Trimmomatic AVGQUAL              : 30
Trimmomatic MINLEN               : 8
Bowtie options                   : --sensitive -L 17
Bowtie index                     : /Users/ferrari/Work/Sandbox/ribomethseq-nf/data/bowtie/human/human_index
samtools options                 : --no-PG -h -u -d 'NM' -e '![XS]'
split count files by rna         : false
Threads for bowtie               : 7
Threads for fastqc               : 2
Threads for trimmomatic          : 3
Threads for samtools             : 4
Export 3'end read count          : false
Log directory                    : /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests
Output directory                 : /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/ribomethseq
Max parallel jobs                : 6

[sample1_R1_001, /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/fastq/sample1_R1_001.fastq.gz]
[sample2_R1_001, /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/fastq/sample2_R1_001.fastq.gz]
[sample3_R1_001, /Users/ferrari/Work/Sandbox/ribomethseq-nf/tests/fastq/sample3_R1_001.fastq.gz]
WARN: Task runtime metrics are not reported when using macOS without a container engine
[96/007339] Submitted process > trim (sample3_R1_001)
[1f/15e562] Submitted process > trim (sample2_R1_001)
[72/cee66f] Submitted process > trim (sample1_R1_001)
[a6/d3ae6b] Submitted process > fastqc (sample3_R1_001)
[30/414a30] Submitted process > fastqc (sample1_R1_001)
[e9/85a734] Submitted process > fastqc (sample2_R1_001)
[c9/fb4438] Submitted process > bowtie2 (sample1_R1_001)
[42/8aa8e1] Submitted process > bowtie2 (sample2_R1_001)
[56/c692ea] Submitted process > filter (sample1_R1_001)
[59/be8e44] Submitted process > bowtie2 (sample3_R1_001)
[30/e47ee1] Submitted process > filter (sample2_R1_001)
[3f/9acffb] Submitted process > counts (sample1_R1_001)
[9e/ad64bc] Submitted process > filter (sample3_R1_001)
[6f/f21824] Submitted process > multiqc
[35/0fb6ef] Submitted process > counts (sample2_R1_001)
[3f/666130] Submitted process > counts (sample3_R1_001)
[81/5634f4] Submitted process > report

Thank you for the helpful suggestion. I followed your advice and decided to delete the ribomethseq-nf files. Afterward, I downloaded them again, and surprisingly, the error disappeared. As a result, I was able to successfully complete most of the steps in the workflow. However, I encountered an issue with the final step, specifically the "report" step. I would greatly appreciate it if you could assist me in resolving this matter.

executor >  local (12)
[d7/a27aed] process > fastqc (878-1_S1_R1)  [100%] 2 of 2 ✔
[8b/e1ff9f] process > trim (879-2_S2_R1)    [100%] 2 of 2 ✔
[2c/ceefec] process > bowtie2 (879-2_S2_R1) [100%] 2 of 2 ✔
[09/047ede] process > filter (879-2_S2_R1)  [100%] 2 of 2 ✔
[33/f41490] process > multiqc               [100%] 1 of 1 ✔
[fa/4c811b] process > counts (878-1_S1_R1)  [100%] 2 of 2 ✔
[-        ] process > split                 -
[44/74ee38] process > report                [  0%] 0 of 1
WARN: Task runtime metrics are not reported when using macOS without a container engine
Error executing process > 'report'

Caused by:
  Process `report` terminated with an error exit status (1)

Command executed:

  Rscript /Users/NAME/Desktop/Ribometh/ribomethseq-nf/Rscripts/QC/generate_report.R .

Command exit status:
  1

the entire log is shown below

6月-13 20:54:56.270 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: fastqc (879-2_S2_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/fb/0b6d09207321043deb2866aa82d3a2]
6月-13 20:54:58.267 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: fastqc (878-1_S1_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/d7/a27aed80361590c4524a5d6353018c]
6月-13 20:56:28.096 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: trim (878-1_S1_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/af/ae61341d12dc17231cad80028b97db]
~> TaskHandler[id: 3; name: trim (879-2_S2_R1); status: RUNNING; exit: -; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/8b/e1ff9fc3c97105d1e4d96cd2ef2967]
~> TaskHandler[id: 5; name: bowtie2 (878-1_S1_R1); status: NEW; exit: -; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/87/5d3b662e49e079f059be966b84fce4]
6月-13 20:58:14.802 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: trim (879-2_S2_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/8b/e1ff9fc3c97105d1e4d96cd2ef2967]
6月-13 21:00:20.528 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 5; name: bowtie2 (878-1_S1_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/87/5d3b662e49e079f059be966b84fce4]
6月-13 21:02:15.620 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 6; name: bowtie2 (879-2_S2_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/2c/ceefec98bbcb044c20ed5f5731c1c6]
~> TaskHandler[id: 7; name: filter (878-1_S1_R1); status: RUNNING; exit: -; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/45/9eba2aeff0bb3e2a3cd55c5a370dce]
~> TaskHandler[id: 8; name: filter (879-2_S2_R1); status: RUNNING; exit: -; error: -; workDir: /Users/uehata/Desktop/Ribometh/ribomethseq-nf/work/09/047ede7b6ad3ed2819ba67be6d02ff]
~> TaskHandler[id: 9; name: multiqc; status: NEW; exit: -; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/33/f41490cc98d9f21b0e14468edcf7e9]
6月-13 21:03:14.727 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 7; name: filter (878-1_S1_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/45/9eba2aeff0bb3e2a3cd55c5a370dce]
6月-13 21:03:14.819 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 8; name: filter (879-2_S2_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/09/047ede7b6ad3ed2819ba67be6d02ff]
6月-13 21:03:39.171 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 9; name: multiqc; status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/33/f41490cc98d9f21b0e14468edcf7e9]
6月-13 21:04:06.686 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 11; name: counts (879-2_S2_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/37/9e90212c7e2d3f3a7909ec4311bc3f]
6月-13 21:04:07.567 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 10; name: counts (878-1_S1_R1); status: COMPLETED; exit: 0; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/fa/4c811b625be9a224f68d1acf36307b]
6月-13 21:09:06.668 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 14; name: report; status: COMPLETED; exit: 1; error: -; workDir: /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/44/74ee382e5f8a96ba735ddc9e0beba4]
6月-13 21:09:06.693 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'report'
  Process `report` terminated with an error exit status (1)
Command error:
  Error in `[.data.frame`:
6月-13 21:09:06.699 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process `report` terminated with an error exit status (1)

Thank you @af8 ! :sunglasses: The most important steps are now working and you should already see some outputs in your outputs directory.

Now, you have a small issue with the last quality control report. Your .nextflow.log shows only the first line of the error :

Command error:
  Error in `[.data.frame`:

But there is another hidden log in the working directory where Rscript failed. You should have the full error message here :

/Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/44/74ee382e5f8a96ba735ddc9e0beba4/.command.log

Can you share this log ?

Apologies for overlooking the fact that the log displayed only a partial error message. Allow me to provide you with the complete error message for better understanding:

次のパッケージを付け加えます: ‘dplyr’ 

 以下のオブジェクトは ‘package:stats’ からマスクされています:

    filter, lag

 以下のオブジェクトは ‘package:base’ からマスクされています:

    intersect, setdiff, setequal, union

Welcome! Want to learn more? See two factoextra-related books at https://goo.gl/ve3WBa
installation directory : /Users/NAME/Desktop/Ribometh/ribomethseq-nf/Rscripts/QC
data directory : /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/44/74ee382e5f8a96ba735ddc9e0beba4
working directory : /Users/NAME/Desktop/Ribometh/ribomethseq-nf/work/44/74ee382e5f8a96ba735ddc9e0beba4

processing file: template.rmd
1/13                   
2/13 [unnamed-chunk-1] 
3/13                   
4/13 [boxplot]         
5/13                   
6/13 [RLE]             
7/13                   
8/13 [heatmap_corr]    
9/13                   
10/13 [heatmap_corr_tri]
11/13                   
12/13 [unnamed-chunk-2] 

Quitting from lines 65-66 [unnamed-chunk-2] (template.rmd)
Error in `[.data.frame`:
!  未定義の列が選ばれました 
Backtrace:
 1. global plot_coa(fcount_matrix)
 2. global coa.plot.col(...)
 4. base::`[.data.frame`(dudi$co, , c(xax, yax))
 実行が停止されました

I would greatly appreciate your assistance in resolving this issue.

Thank you ! I have managed to reproduce this error.

This is because you have only two samples, which makes the correspondence analysis fail at the end of the quality control report.

If you have one more sample, it should work.

I will work on a small fix to prevent the report step from crashing when there are two or even one samples.

RibosomeCRCL / ribomethseq-nf