Error executing process > 'concatenateFastq'

[isa-wilson]

Keen to use this super helpful pipeline but stumbling at the first hurdle.

When I run nextflow -c metontiime2_GIIIA.conf run metontiime2.nf --workDir="/Users/a1706564/ONT_wkdir/pool_1/fastq_pass" --resultsDir="/Users/a1706564/ONT_wkdir/pool_1/resultsDir" -profile docker, the 'concatenateFastq' process keeps failing:

ERROR ~ Error executing process > 'concatenateFastq'

Caused by:
  Process `concatenateFastq` terminated with an error exit status (1)

Command executed:

  mkdir -p /Users/a1706564/ONT_wkdir/pool_1/resultsDir
  mkdir -p /Users/a1706564/ONT_wkdir/pool_1/resultsDir/concatenateFastq

  barcodes_dirs=$(realpath $(find /Users/a1706564/ONT_wkdir/pool_1/fastq_pass -maxdepth 1 -mindepth 1 | grep "barcode"));
  echo $barcodes_dirs
  for b in $barcodes_dirs; do
  (emptybn=$(basename $b);
    f=$(find $b | grep ".fastq");
  mmand zless $f | gzip > /Users/a1706564/ONT_wkdir/pool_1/resultsDir/concatenateFastq/$bn.fastq.gz;
  done

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  find: ‘/Users/a1706564/ONT_wkdir/pool_1/fastq_pass’: No such file or directory
  realpath: missing operand
  Try 'realpath --help' for more information.

Seems like it's having issues identifying the fastq.gz files in my working directory? The working directory contains a series of folders (barcodeXX) containing my already demultiplexed reads & I have set "concatenateFastq" to "true" in the conf file.

I'm sure I'm overlooking something obvious! I've attached the conf file in case that helps.

metontiime2_GIIIA.conf.txt

[MaestSi]

Hi, you should make accessible /Users directory to the Docker image. To do that, just edit line 173 of the conf file from: containerOptions = '-v /home/:/home' to: containerOptions = '-v /Users:/Users' Let me know if this solves the issue. Best, SM

[isa-wilson]

It worked - thank you so much!

Unfortunately I am now having issues with importDb:

ERROR ~ Error executing process > 'importDb (1)'

Caused by:
  Process `importDb (1)` terminated with an error exit status (132)

Command executed:

  mkdir -p /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb

  qiime tools import        --type 'FeatureData[Sequence]'      --input-path SILVA_138.1_SSURef_NR99_tax_silva.fasta        --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_sequences.qza

  qiime tools import        --type 'FeatureData[Taxonomy]'      --input-path taxmap_slv_ssu_ref_nr_138.1.tsv        --input-format HeaderlessTSVTaxonomyFormat      --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_taxonomy.qza

Command exit status:
  132

Command output:
  (empty)

Command error:
  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  .command.sh: line 4:     8 Illegal instruction     qiime tools import --type 'FeatureData[Sequence]' --input-path SILVA_138.1_SSURef_NR99_tax_silva.fasta --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_sequences.qza

I have used the full pathnames to the seq and tax database files in the conf file. Is the issue that I'm trying to run this on an M1 Mac? I've ticked "Use Rosetta for x86_64/amd64 emulation on Apple Silicon" in Docker Desktop...

[MaestSi]

Hi, can you please go to the work directory corresponding to importDb process (you should be able to find the subfolder directory name, inside the work dir, based on the process hash), and check that in that folder there is a symbolic link for SILVA_138.1_SSURef_NR99_tax_silva.fasta, pointing to its full path? Moreover, is the full path inside /Users folder? If that is not the case, for example because you have fasta and taxonomy files on a separate hard drive, you should also mount that disk with: containerOptions = '-v /Users:/Users -v /hardDrivePath:/hardDrivePath' If this does not solve the issue, could you please send me your config file and the exact command line that you ran? Best, SM

[isa-wilson]

Hi, the work dir (i.e. f9/8885c5 in the below error message) contains symbolic links for both the .fasta and .tsv database files pointing to their respective full paths. The importDb folder (within resultsDir) is empty.

Everything I'm working with is within /Users so I don't think that is the issue.

I used the following command: nextflow -c metontiime2_GIIIA.conf run metontiime2.nf --workDir="/Users/a1706564/ONT_wkdir/pool_1/fastq_pass" --resultsDir="/Users/a1706564/ONT_wkdir/pool_1/resultsDir" -profile docker

Here is my config file: metontiime2_GIIIA.conf.txt

And here is the full error message in case it is useful:

Nextflow 24.04.3 is available - Please consider updating your version to it

 N E X T F L O W   ~  version 24.04.2

Launching `metontiime2.nf` [special_faraday] DSL2 - revision: ce81e587ac

WARN: Killing running tasks (1)
executor >  local (3)
[f9/8885c5] importDb (1)          | 0 of 1
[92/5fbd6c] concatenateFastq      | 1 of 1 ✔
[c9/9adec1] filterFastq           | 0 of 1
[-        ] downsampleFastq       -
[-        ] importFastq           -
[-        ] derepSeq              -
[-        ] assignTaxonomy        -
[-        ] filterTaxa            -
[-        ] taxonomyVisualization -
[-        ] collapseTables        -
[-        ] dataQC                -
[-        ] diversityAnalyses     -
ERROR ~ Error executing process > 'importDb (1)'

Caused by:
  Process `importDb (1)` terminated with an error exit status (132)

Command executed:

  mkdir -p /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb

  qiime tools import        --type 'FeatureData[Sequence]'      --input-path SILVA_138.1_SSURef_NR99_tax_silva.fasta        --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_sequences.qza

  qiime tools import        --type 'FeatureData[Taxonomy]'      --input-path taxmap_slv_ssu_ref_nr_138.1.tsv        --input-format HeaderlessTSVTaxonomyFormat      --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDiexecutor >  local (3)
[f9/8885c5] importDb (1)          | 1 of 1, failed: 1 ✘
[92/5fbd6c] concatenateFastq      | 1 of 1 ✔
[-        ] filterFastq           -
[-        ] downsampleFastq       -
[-        ] importFastq           -
[-        ] derepSeq              -
[-        ] assignTaxonomy        -
[-        ] filterTaxa            -
[-        ] taxonomyVisualization -
[-        ] collapseTables        -
[-        ] dataQC                -
[-        ] diversityAnalyses     -
ERROR ~ Error executing process > 'importDb (1)'

Caused by:
  Process `importDb (1)` terminated with an error exit status (132)

Command executed:

  mkdir -p /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb

  qiime tools import        --type 'FeatureData[Sequence]'      --input-path SILVA_138.1_SSURef_NR99_tax_silva.fasta        --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_sequences.qza

  qiime tools import        --type 'FeatureData[Taxonomy]'      --input-path taxmap_slv_ssu_ref_nr_138.1.tsv        --input-format HeaderlessTSVTaxonomyFormat      --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_taxonomy.qza

Command exit status:
  132

Command output:
  (empty)

Command error:
  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  .command.sh: line 4:     8 Illegal instruction     qiime tools import --type 'FeatureData[Sequence]' --input-path SILVA_138.1_SSURef_NR99_tax_silva.fasta --output-path /Users/a1706564/ONT_wkdir/pool_1/resultsDir/importDb/db_sequences.qza

Work dir:
  /Users/a1706564/MetONTIIME/work/f9/8885c58f21d03acfe46b98494eee7d

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Here is my config file: metontiime2_GIIIA.conf.txt

[MaestSi]

Hi, first of all, I noticed sampleMetadata parameter does not contain the full path to the file, while it should. The file may not exist yet. If that is the case, it is created at runtime exactly where you set it in the corresponding parameter. I didn't notice anything else possibly wrong in the config file. SM

[MaestSi]

I remember a user previously reported a similar issue. You may have to adjust Docker configuration to solve this. In this post, I noticed the user had to enable "Use Rosetta for x86/amd64 emulation on Apple Silicon", this suggestion may be helpful. SM

[isa-wilson]

Yep, looks like it is the same issue. I have had "Use Rosetta for x86/amd64 emulation on Apple Silicon" enabled this whole time so it looks like I will have to play around with Docker a bit more. I'll post here if I end up finding a fix :)

Thanks again for all of your help.