Could not run test using Docker

[yuifu]

Description of the bug

When I run test for genomeannotator, I got the following errror:

Error executing process > 'NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)'

Caused by:
  Process `NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)` terminated with an error exit status (1)

Command executed:

  mkdir -p augustus_config
  cp -R config/* augustus_config/

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG":
      augustus: $(echo $(augustus  | head -n1 | cut -f2 -d " " | sed "s/[)]//" | sed "s/[(]//" ))
  END_VERSIONS

Command output:
  (empty)

Command error:
  Unable to find image 'quay.io/biocontainers/augustus:3.4.0--pl5262h5a9fe7b_2' locally
  3.4.0--pl5262h5a9fe7b_2: Pulling from biocontainers/augustus
  c1a16a04cedd: Already exists
  4ca545ee6d5d: Already exists
  9045f8e140e3: Pulling fs layer
  9045f8e140e3: Verifying Checksum
  9045f8e140e3: Download complete
  9045f8e140e3: Pull complete
  Digest: sha256:db15c7f970ce92b36b5f657062bd948634d0c4c162b21fc67916b6ba23ea1770
  Status: Downloaded newer image for quay.io/biocontainers/augustus:3.4.0--pl5262h5a9fe7b_2
  cp: can't stat 'config/*': No such file or directory

Command used and terminal output

$ nextflow run nf-core/genomeannotator -r dev -profile test,docker --outdir genomeannotator_test_001

Relevant files

No response

System information

• Nextflow version version 22.04.4 build 5706
• executor: local
• Hardware: HPC
• Container engine: Docker
• OS: CentOS Linux 7

[cgrard]

I'm having a similar issue however it is slightly different even though the result is the same :

ERROR ~ Error executing process > 'NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)'

Caused by:
  Process `NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)` terminated with an error exit status (127)

Command executed:

  mkdir -p augustus_config
  cp -R config/* augustus_config/

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG":
      augustus: $(echo $(augustus  | head -n1 | cut -f2 -d " " | sed "s/[)]//" | sed "s/[(]//" ))
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  Unable to find image 'quay.io/biocontainers/augustus:3.4.0--pl5262h5a9fe7b_2' locally
  3.4.0--pl5262h5a9fe7b_2: Pulling from biocontainers/augustus
  c1a16a04cedd: Already exists
  4ca545ee6d5d: Already exists
  9045f8e140e3: Pulling fs layer
  9045f8e140e3: Verifying Checksum
  9045f8e140e3: Download complete
  9045f8e140e3: Pull complete
  Digest: sha256:db15c7f970ce92b36b5f657062bd948634d0c4c162b21fc67916b6ba23ea1770
  Status: Downloaded newer image for quay.io/biocontainers/augustus:3.4.0--pl5262h5a9fe7b_2
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/env-execute": stat /usr/local/env-execute: no such file or directory: unknown.

I'm using the same command on the same kind of system except it is Ubuntu 20.04 and not CentOS 7

[elizabethmcd]

I think I have an identical issue as @cgrard, except running on MacOS with Docker and Nextflow version 22.10.6 build 5843:

Error executing process > 'NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)'

Caused by:
  Process `NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG (config)` terminated with an error exit status (127)

Command executed:

  mkdir -p augustus_config
  cp -R config/* augustus_config/

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:AUGUSTUS_STAGECONFIG":
      augustus: $(echo $(augustus  | head -n1 | cut -f2 -d " " | sed "s/[)]//" | sed "s/[(]//" ))
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/usr/local/env-execute": stat /usr/local/env-execute: no such file or directory: unknown.
  time="2023-08-24T11:21:09-07:00" level=error msg="error waiting for container: context canceled"

Ran with command nextflow run nf-core/genomeannotator -profile test,docker --outdir test_annotator -r dev

[elizabethmcd]

Update, I cloned the most recent pipeline from @marchoeppner dev branch, where I had to change # to // in the modules/local/spaln/align/main.nf script, and no longer get the above error from that fork. Most of the steps complete fine but have a new error at the CUSTOM_DUMPSOFTWAREVERSIONS step:

Error executing process > 'NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:CUSTOM_DUMPSOFTWAREVERSIONS (1)'

Caused by:
  Process `NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:CUSTOM_DUMPSOFTWAREVERSIONS (1)` terminated with an error exit status (1)

Command executed [/Users/emcdaniel/Desktop/Arcadia/workflows/genomeannotator/./workflows/../modules/nf-core/custom/dumpsoftwareversions/templates/dumpsoftwareversions.py]:

  #!/usr/bin/env python

  """Provide functions to merge multiple versions.yml files."""

  import yaml
  import platform
  from textwrap import dedent

  def _make_versions_html(versions):
      """Generate a tabular HTML output of all versions for MultiQC."""
      html = [
          dedent(
              """\
              <style>
              #nf-core-versions tbody:nth-child(even) {
                  background-color: #f2f2f2;
              }
              </style>
              <table class="table" style="width:100%" id="nf-core-versions">
                  <thead>
                      <tr>
                          <th> Process Name </th>
                          <th> Software </th>
                          <th> Version  </th>
                      </tr>
                  </thead>
              """
          )
      ]
      for process, tmp_versions in sorted(versions.items()):
          html.append("<tbody>")
          for i, (tool, version) in enumerate(sorted(tmp_versions.items())):
              html.append(
                  dedent(
                      f"""\
                      <tr>
                          <td><samp>{process if (i == 0) else ''}</samp></td>
                          <td><samp>{tool}</samp></td>
                          <td><samp>{version}</samp></td>
                      </tr>
                      """
                  )
              )
          html.append("</tbody>")
      html.append("</table>")
      return "\n".join(html)

  def main():
      """Load all version files and generate merged output."""
      versions_this_module = {}
      versions_this_module["NFCORE_GENOMEANNOTATOR:GENOMEANNOTATOR:CUSTOM_DUMPSOFTWAREVERSIONS"] = {
          "python": platform.python_version(),
          "yaml": yaml.__version__,
      }

      with open("collated_versions.yml") as f:
          versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module

      # aggregate versions by the module name (derived from fully-qualified process name)
      versions_by_module = {}
      for process, process_versions in versions_by_process.items():
          module = process.split(":")[-1]
          try:
              if versions_by_module[module] != process_versions:
                  raise AssertionError(
                      "We assume that software versions are the same between all modules. "
                      "If you see this error-message it means you discovered an edge-case "
                      "and should open an issue in nf-core/tools. "
                  )
          except KeyError:
              versions_by_module[module] = process_versions

      versions_by_module["Workflow"] = {
          "Nextflow": "22.10.6",
          "nf-core/genomeannotator": "1.0dev",
      }

      versions_mqc = {
          "id": "software_versions",
          "section_name": "nf-core/genomeannotator Software Versions",
          "section_href": "https://github.com/nf-core/genomeannotator",
          "plot_type": "html",
          "description": "are collected at run time from the software output.",
          "data": _make_versions_html(versions_by_module),
      }

      with open("software_versions.yml", "w") as f:
          yaml.dump(versions_by_module, f, default_flow_style=False)
      with open("software_versions_mqc.yml", "w") as f:
          yaml.dump(versions_mqc, f, default_flow_style=False)

      with open("versions.yml", "w") as f:
          yaml.dump(versions_this_module, f, default_flow_style=False)

  if __name__ == "__main__":
      main()

Command exit status:
  1

Command output:
  (empty)

Command error:
  Traceback (most recent call last):
    File ".command.sh", line 101, in <module>
      main()
    File ".command.sh", line 61, in main
      versions_by_process = yaml.load(f, Loader=yaml.BaseLoader) | versions_this_module
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
      return loader.get_single_data()
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
      node = self.get_single_node()
             ^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
      document = self.compose_document()
                 ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
      node = self.compose_node(None, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
      item_value = self.compose_node(node, item_key)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
      node = self.compose_mapping_node(anchor)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/composer.py", line 127, in compose_mapping_node
      while not self.check_event(MappingEndEvent):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
      self.current_event = self.state()
                           ^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/parser.py", line 428, in parse_block_mapping_key
      if self.check_token(KeyToken):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 116, in check_token
      self.fetch_more_tokens()
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
      return self.fetch_value()
             ^^^^^^^^^^^^^^^^^^
    File "/usr/local/lib/python3.11/site-packages/yaml/scanner.py", line 577, in fetch_value
      raise ScannerError(None, None,
  yaml.scanner.ScannerError: mapping values are not allowed here
    in "collated_versions.yml", line 21, column 13

Will try to investigate the issue since it looks like the pipeline might be nearing a v1 and a great tool for the community!

[marchoeppner]

Hi Elizabeth,

thanks for looking into this. Most likely one of the many steps emits a slightly malformed version block. You should be able to spot which when you look into the collated versions yml file inside your work directory.

I have just run the basic built-in test on my local code, and that completes without issues. Can you try the most recent version from my dev branch to see if my latest push fixed the issue?

Cheers,

Marc

[marchoeppner]

As for the docker issue - apologies, haven't kept an eye on this repo while I was working on other stuff.

The message suggests that your docker demon is, for some reason, unable to reach the biocontainers repo. The container does exists, as far as I can tell, so I am not sure why it wouldn't work. Maybe you can try to clone the container outside of the pipeline to check that docker and docker pull work as expected.

[elizabethmcd]

Success with the latest pull from your dev branch!

-[nf-core/genomeannotator] Pipeline completed successfully-
Completed at: 25-Aug-2023 13:17:00
Duration    : 1h 7m 23s
CPU hours   : 5.3
Succeeded   : 77

[marchoeppner]

Great - so presumably the original issue with Docker is also solved? If not, please reopen and tag me in the update.