search

nf-core / ampliseq

Amplicon sequencing analysis workflow using DADA2 and QIIME2
https://nf-co.re/ampliseq
MIT License
187 stars 117 forks source link

Error when running the pipeline on a machine with Apple silicon (M1 processor) #524

Closed gbaiao closed 1 year ago

gbaiao commented 1 year ago

Hi! I have run the ampliseq pipeline before on an older macbook without any problems, but now I'm tryin to run it on a newer computer equipped with an Apple M1 processor and it seems to be running into some issue.

This is the error I get when I run the test dataset:

nextflow run nf-core/ampliseq -profile test,docker --outdir ./

-[nf-core/ampliseq] Pipeline completed with errors-
Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_DIVERSITY_CORE (1)'

Caused by:
  Process `NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_DIVERSITY_CORE (1)` terminated with an error exit status (1)

Command executed:

  export XDG_CONFIG_HOME="${PWD}/HOME"

  mindepth=$(count_table_minmax_reads.py filtered-table.tsv minimum 2>&1)
  if [ "$mindepth" -gt "10000" ]; then echo $mindepth >"Use the sampling depth of $mindepth for rarefaction.txt" ; fi
  if [ "$mindepth" -lt "10000" -a "$mindepth" -gt "5000" ]; then echo $mindepth >"WARNING The sampling depth of $mindepth is quite small for rarefaction.txt" ; fi
  if [ "$mindepth" -lt "5000" -a "$mindepth" -gt "1000" ]; then echo $mindepth >"WARNING The sampling depth of $mindepth is very small for rarefaction.txt" ; fi
  if [ "$mindepth" -lt "1000" ]; then echo $mindepth >"WARNING The sampling depth of $mindepth seems too small for rarefaction.txt" ; fi

  qiime diversity core-metrics-phylogenetic \
      --m-metadata-file Metadata.tsv \
      --i-phylogeny rooted-tree.qza \
      --i-table filtered-table.qza \
      --p-sampling-depth $mindepth \
      --output-dir diversity_core \
      --p-n-jobs-or-threads 2 \
      --verbose

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_DIVERSITY_CORE":
      qiime2: $( qiime --version | sed '1!d;s/.* //' )
  END_VERSIONS

Command exit status:
  1

Command output:
  Running external command line application. This may print messages to stdout and/or stderr.
  The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

  Command:

  faithpd -i /tmp/qiime2/uid=#501/data/3804c417-e5bc-440b-afba-831560f01f7c/data/feature-table.biom -t /tmp/qiime2/uid=#501/data/65fb7197-964b-4210-a999-adb203415552/data/tree.nwk -o /tmp/q2-AlphaDiversityFormat-tsebz08o

Command error:
  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
  qemu: uncaught target signal 11 (Segmentation fault) - core dumped
  /opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/sklearn/metrics/pairwise.py:1776: DataConversionWarning: Data was converted to boolean for metric jaccard
    warnings.warn(msg, DataConversionWarning)
  Traceback (most recent call last):
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__
      results = action(**arguments)
    File "<decorator-gen-117>", line 2, in core_metrics_phylogenetic
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
      outputs = self._callable_executor_(scope, callable_args,
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 475, in _callable_executor_
      outputs = self._callable(scope.ctx, **view_args)
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity/_core_metrics.py", line 62, in core_metrics_phylogenetic
      faith_pd_vector, = faith_pd(table=cr.rarefied_table,
    File "<decorator-gen-470>", line 2, in faith_pd
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
      outputs = self._callable_executor_(scope, callable_args,
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in _callable_executor_
      output_views = self._callable(**view_args)
    File "<decorator-gen-71>", line 2, in faith_pd
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 69, in _disallow_empty_tables
      return wrapped_function(*args, **kwargs)
    File "<decorator-gen-70>", line 2, in faith_pd
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 112, in _validate_requested_cpus
      return wrapped_function(*bound_arguments.args, **bound_arguments.kwargs)
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/alpha.py", line 53, in faith_pd
      _omp_cmd_wrapper(threads, cmd)
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 128, in _omp_cmd_wrapper
      return _run_external_cmd(cmd, verbose=verbose, env=env)
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_diversity_lib/_util.py", line 122, in _run_external_cmd
      return subprocess.run(cmd, check=True, env=env)
    File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['faithpd', '-i', '/tmp/qiime2/uid=#501/data/3804c417-e5bc-440b-afba-831560f01f7c/data/feature-table.biom', '-t', '/tmp/qiime2/uid=#501/data/65fb7197-964b-4210-a999-adb203415552/data/tree.nwk', '-o', '/tmp/q2-AlphaDiversityFormat-tsebz08o']' died with <Signals.SIGSEGV: 11>.

  Plugin error from diversity:

    Command '['faithpd', '-i', '/tmp/qiime2/uid=#501/data/3804c417-e5bc-440b-afba-831560f01f7c/data/feature-table.biom', '-t', '/tmp/qiime2/uid=#501/data/65fb7197-964b-4210-a999-adb203415552/data/tree.nwk', '-o', '/tmp/q2-AlphaDiversityFormat-tsebz08o']' died with <Signals.SIGSEGV: 11>.

  See above for debug info.
  Running external command line application. This may print messages to stdout and/or stderr.
  The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.

  Command:

  faithpd -i /tmp/qiime2/uid=#501/data/3804c417-e5bc-440b-afba-831560f01f7c/data/feature-table.biom -t /tmp/qiime2/uid=#501/data/65fb7197-964b-4210-a999-adb203415552/data/tree.nwk -o /tmp/q2-AlphaDiversityFormat-tsebz08o

Work dir:
  /Users/gbaiao/Softwares/AmpliSeq/work/8d/36954d4fadfb52e430d720c3a9cd86

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

If you think I may be doing something wrong, please let me know. Best,

d4straub commented 1 year ago

Yes, the M1/M2 chips of Mac arent compatible with docker and singularity as far as I know. Your older Mac had most likely an Intel chipset. I have myself no experience with M1/2, but colleagues do not develop and run container-based pipelines with M1/2 but use remote Linux workstations or our compute cluster. As far as I understood M1/2 is not compatible with docker/singularity, it is unbearable slow. Maybe in the future that will change, but currently, as far as I heared, executing nf-core pipelines with M1/2 isnt working.

This is also in your error message: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)

gbaiao commented 1 year ago

Ah, I see. Looking on Docker’s website, it seem like they have a version designed for M1/M2 processor (https://www.docker.com/blog/released-docker-desktop-for-mac-apple-silicon/) but perhaps it’s still very slow? If there is indeed a version for M1/M2, would it be possible for me to somehow adjust the pipeline for running on these processors? Or is that something that would have to be done on the developer’s side? Please bear in mind that this is my first time using Docker, so I don’t know if I may be asking something that doesn't make completete sense...

On 23 Jan 2023, at 05:14, Daniel Straub @.***> wrote:

Yes, the M1/M2 chips of Mac arent compatible with docker and singularity as far as I know. Your older Mac had most likely an Intel chipset. I have myself no experience with M1/2, but colleagues do not develop and run container-based pipelines with M1/2 but use remote Linux workstations or our compute cluster. As far as I understood M1/2 is not compatible with docker/singularity, it is unbearable slow. Maybe in the future that will change, but currently, as far as I heared, executing nf-core pipelines with M1/2 isnt working.

— Reply to this email directly, view it on GitHub https://github.com/nf-core/ampliseq/issues/524#issuecomment-1399951319, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYBIIX32U4JOLNWUOCDIGLWTY4XJANCNFSM6AAAAAAUB34TEI. You are receiving this because you authored the thread.

d4straub commented 1 year ago

it seem like they have a version designed for M1/M2 processor (https://www.docker.com/blog/released-docker-desktop-for-mac-apple-silicon/) but perhaps it’s still very slow? If there is indeed a version for M1/M2, would it be possible for me to somehow adjust the pipeline for running on these processors?

Most recent colleague started end of 2022 and couldnt get it to run, so I dont think it works.

Or is that something that would have to be done on the developer’s side?

I think that has to be first solved by docker, then we could add support to nf-core pipelines if it requires any adjustments. But (nf-core) pipelines are targeting workstations/hpc/cloud, and the M1/2 chips arent really used in that environments afaik, laptops are just for tests or development, so I do not think this is high priority.

Please bear in mind that this is my first time using Docker, so I don’t know if I may be asking something that doesn't make completete sense...

No problem

gbaiao commented 1 year ago

Ok, it makes sense. Thank you very much!

On 23 Jan 2023, at 12:08, Daniel Straub @.***> wrote:

it seem like they have a version designed for M1/M2 processor (https://www.docker.com/blog/released-docker-desktop-for-mac-apple-silicon/) but perhaps it’s still very slow? If there is indeed a version for M1/M2, would it be possible for me to somehow adjust the pipeline for running on these processors?

Most recent colleague started end of 2022 and couldnt get it to run, so I dont think it works.

Or is that something that would have to be done on the developer’s side?

I think that has to be first solved by docker, then we could add support to nf-core pipelines if it requires any adjustments. But (nf-core) pipelines are targeting workstations/hpc/cloud, and the M1/2 chips arent really used in that environments afaik, laptops are just for tests or development, so I do not think this is high priority.

Please bear in mind that this is my first time using Docker, so I don’t know if I may be asking something that doesn't make completete sense...

No problem

— Reply to this email directly, view it on GitHub https://github.com/nf-core/ampliseq/issues/524#issuecomment-1400505766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYBIIV653AYTTFSH44JWJTWT2NHNANCNFSM6AAAAAAUB34TEI. You are receiving this because you authored the thread.

d4straub commented 1 year ago

I close that issue for now, since solving that issue isn't really something we can solve for now. Please feel free to open another issue when you have questions or when you come across any solution to the problem.

d4straub commented 1 year ago

Update: -profile conda might work, at least mostly, and without QIIME2.