Pipeline finishes with no plots/analysis with qiime2

skose82 commented 1 year ago

Description of the bug

Pipeline finishes with no plots/analysis with qiime2 (e.g. ancom, barplot, rarefacation_tables) only taxonomy folder present. The software dump process fails but assume plots were completed before this step. Command run with conda.

Command used and terminal output

[b8/e9b36e] process > NFCORE_AMPLISEQ:AMPLISEQ:RENAME_RAW_DATA_FILES (S103)   [100%] 4 of 4 ✔
[45/910613] process > NFCORE_AMPLISEQ:AMPLISEQ:FASTQC (S103)                  [100%] 4 of 4 ✔
[40/a69141] process > NFCORE_AMPLISEQ:AMPLISEQ:CUTADAPT_WORKFLOW:CUTADAPT_... [100%] 4 of 4 ✔
[b5/826cf4] process > NFCORE_AMPLISEQ:AMPLISEQ:CUTADAPT_WORKFLOW:CUTADAPT_... [100%] 1 of 1 ✔
[6e/2e8c62] process > NFCORE_AMPLISEQ:AMPLISEQ:CUTADAPT_WORKFLOW:CUTADAPT_... [100%] 1 of 1 ✔
[46/de743c] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_PREPROCESSING:DADA2_Q... [100%] 2 of 2 ✔
[1c/d23598] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_PREPROCESSING:TRUNCLE... [100%] 2 of 2 ✔
[52/e64be2] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_PREPROCESSING:DADA2_F... [100%] 4 of 4 ✔
[14/4edfe6] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_PREPROCESSING:DADA2_Q... [100%] 2 of 2 ✔
[f7/86d28d] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_ERR (1)                  [100%] 1 of 1 ✔
[b0/01ae96] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_DENOISING (1)            [100%] 1 of 1 ✔
[dd/ff41cb] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_RMCHIMERA (1)            [100%] 1 of 1 ✔
[6d/d480b4] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_STATS (1)                [100%] 1 of 1 ✔
[40/17d888] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_MERGE                    [100%] 1 of 1 ✔
[9b/8e4c79] process > NFCORE_AMPLISEQ:AMPLISEQ:MERGE_STATS_STD (1)            [100%] 1 of 1 ✔
[76/fb146a] process > NFCORE_AMPLISEQ:AMPLISEQ:BARRNAP (ASV_seqs.fasta)       [100%] 1 of 1 ✔
[ae/594ef1] process > NFCORE_AMPLISEQ:AMPLISEQ:FORMAT_TAXONOMY                [100%] 1 of 1 ✔
[9b/fec333] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_TAXONOMY (ASV_seqs.fa... [100%] 1 of 1 ✔
[e1/55e6e7] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_ADDSPECIES (ASV_tax.r... [100%] 1 of 1 ✔
[73/6612d8] process > NFCORE_AMPLISEQ:AMPLISEQ:CUSTOM_DUMPSOFTWAREVERSIONS... [  0%] 0 of 1
[-        ] process > NFCORE_AMPLISEQ:AMPLISEQ:MULTIQC                        -
Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
  Process `NFCORE_AMPLISEQ:AMPLISEQ:CUSTOM_DUMPSOFTWAREVERSIONS (1)` terminated with an error exit status (1)

Command executed [/Users/user1/.nextflow/assets/nf-core/ampliseq/./workflows/../modules/nf-core/custom/dumpsoftwareversions/templates/dumpsoftwareversions.py]:

  #!/usr/bin/env python

  import yaml
  import platform
  from textwrap import dedent

  def _make_versions_html(versions):
      html = [
          dedent(
              """\
              <style>
              #nf-core-versions tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions">
                  <thead>
                      <tr>
                          <th> Process Name </th>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              """
          )
      ]
      for process, tmp_versions in sorted(versions.items()):
          html.append("<tbody>")
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              html.append(
                  dedent(
                      f"""\
                      <tr>
                          <td><samp>{process if (i == 0) else ''}</samp></td>
                          <td><samp>{tool}</samp></td>
                          <td><samp>{version}</samp></td>
                      </tr>
                      """
                  )
              )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)

  versions_this_module = {}
  versions_this_module["NFCORE_AMPLISEQ:AMPLISEQ:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
      "python": platform.python_version(),
      "yaml": yaml.__version__,
  }

  with open("collated_versions.yml") as f:
      versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module

  # aggregate versions by the module name (derived from fully-qualified process name)
  versions_by_module = {}
  for process, process_versions in versions_by_process.items():
      module = process.split(":")[-1]
      try:
          assert versions_by_module[module] == process_versions, (
              "We assume that software versions are the same between all modules. "
              "If you see this error-message it means you discovered an edge-case "
              "and should open an issue in nf-core/tools. "
          )
      except KeyError:
          versions_by_module[module] = process_versions

  versions_by_module["Workflow"] = {
      "Nextflow": "22.10.4",
      "nf-core/ampliseq": "2.4.1",
  }

  versions_mqc = {
      "id": "software_versions",
      "section_name": "nf-core/ampliseq Software Versions",
      "section_href": "https://github.com/nf-core/ampliseq",
      "plot_type": "html",
      "description": "are collected at run time from the software output.",
      "data": _make_versions_html(versions_by_module),
  }

  with open("software_versions.yml", "w") as f:
      yaml.dump(versions_by_module, f, default_flow_style=False)
  with open("software_versions_mqc.yml", "w") as f:
      yaml.dump(versions_mqc, f, default_flow_style=False)

  with open("versions.yml", "w") as f:
      yaml.dump(versions_this_module, f, default_flow_style=False)

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File ".command.sh", line 54, in <module>
      versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module
    File "/Users/user1/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load
      return loader.get_single_data()
    File "/Users/user1/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
    File "/Users/user1/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/yaml/composer.py", line 36, in get_single_node
      document = self.compose_document()
    File "/Users/user1/opt/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/yaml/composer.py", line 55, in compose_document
      node = self.compose_node(None, None)

Relevant files

No response

System information

nextflow 22.10.4 nf-core/ampliseq v2.4.1 Desktop MacOS/M2 Conda

d4straub commented 1 year ago

M1/M2 isnt really supported, but it would be certainly great if the pipeline would work on that as well. (nf-core) pipelines are targeting workstations/hpc/cloud, and the M1/2 chips arent really used in that environments afaik, laptops are just for tests or development. Conda should work mostly on M1/M2 chips, as it did for you.

Pipeline finishes with no plots/analysis with qiime2 (e.g. ancom, barplot, rarefacation_tables) only taxonomy folder present.

However, QIIME2 has only a docker image, no conda package, so with conda there cannot be and QIIME2 ouptut, so thats not a bug.

The software dump process fails but assume plots were completed before this step.

The process CUSTOM_DUMPSOFTWAREVERSIONS comes straight from the nf-core template (even if it contains "custom"). Probably it would be best to discuss this in the nf-core slack channel help to get more knowledgeable people in that matter to help you?

d4straub commented 1 year ago

I'll close that issue, please feel free to open other issues if needed (and you did already).

nf-core / ampliseq