Demo fails (on Mac)

[phiweger]

I installed charcoal by the book and tried to run the demo as documented in the README. Alas!

python -m charcoal run demo/demo.conf -j 4
** read 2 provided lineages
** config file checks PASSED!
** from here on out, it's all snakemake...
Building DAG of jobs...
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Creating conda environment charcoal/conf/env-sourmash.yml...
Downloading and installing remote packages.
Environment for charcoal/conf/env-sourmash.yml created (location: .snakemake/conda/16fae472)
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Creating conda environment charcoal/conf/env-reporting.yml...
Downloading and installing remote packages.
Environment for charcoal/conf/env-reporting.yml created (location: .snakemake/conda/2f61c56c)
Using shell: /bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    1   combine_genome_summary
    1   combine_hit_list
    6   compare_taxonomy_single
    6   contigs_sig
    6   make_contigs_taxonomy_json
    1   make_index
    6   search_all
    1   set_kernel
    29

[Mon Jan 11 16:53:52 2021]
rule contigs_sig:
    input: demo/genomes/TARA_PON_MAG_00084.fa.gz
    output: output.demo/stage1/TARA_PON_MAG_00084.fa.gz.sig
    jobid: 21
    wildcards: filename=TARA_PON_MAG_00084.fa.gz

[Mon Jan 11 16:53:52 2021]
rule contigs_sig:
    input: demo/genomes/LoombaR_2017__SID1050_bax__bin.11.fa.gz
    output: output.demo/stage1/LoombaR_2017__SID1050_bax__bin.11.fa.gz.sig
    jobid: 9
    wildcards: filename=LoombaR_2017__SID1050_bax__bin.11.fa.gz

[Mon Jan 11 16:53:52 2021]
rule contigs_sig:
    input: demo/genomes/TOBG_NAT-167.fna.gz
    output: output.demo/stage1/TOBG_NAT-167.fna.gz.sig
    jobid: 13
    wildcards: filename=TOBG_NAT-167.fna.gz

[Mon Jan 11 16:53:52 2021]
rule contigs_sig:
    input: demo/genomes/GCA_001593925.1_ASM159392v1_genomic.fna.gz
    output: output.demo/stage1/GCA_001593925.1_ASM159392v1_genomic.fna.gz.sig
    jobid: 25
    wildcards: filename=GCA_001593925.1_ASM159392v1_genomic.fna.gz

WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.compat module is deprecated and will be removed in a future release.
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Activating conda environment: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
Activating conda environment: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
Activating conda environment: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
WARNING: The conda.compat module is deprecated and will be removed in a future release.
/bin/bash: sourmash: command not found
/bin/bash: sourmash: command not found
/bin/bash: sourmash: command not found
[Mon Jan 11 16:53:53 2021]
Error in rule contigs_sig:
[Mon Jan 11 16:53:53 2021]
    jobid: 13
Error in rule contigs_sig:
[Mon Jan 11 16:53:53 2021]
    output: output.demo/stage1/TOBG_NAT-167.fna.gz.sig
    jobid: 21
Error in rule contigs_sig:
    conda-env: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
    output: output.demo/stage1/TARA_PON_MAG_00084.fa.gz.sig
    jobid: 9
    shell:

        sourmash compute -k 31 --scaled 1000             --dna demo/genomes/TOBG_NAT-167.fna.gz -o output.demo/stage1/TOBG_NAT-167.fna.gz.sig

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    conda-env: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
    output: output.demo/stage1/LoombaR_2017__SID1050_bax__bin.11.fa.gz.sig

    shell:

        sourmash compute -k 31 --scaled 1000             --dna demo/genomes/TARA_PON_MAG_00084.fa.gz -o output.demo/stage1/TARA_PON_MAG_00084.fa.gz.sig

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    conda-env: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472

    shell:

        sourmash compute -k 31 --scaled 1000             --dna demo/genomes/LoombaR_2017__SID1050_bax__bin.11.fa.gz -o output.demo/stage1/LoombaR_2017__SID1050_bax__bin.11.fa.gz.sig

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Activating conda environment: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
/bin/bash: sourmash: command not found
[Mon Jan 11 16:53:54 2021]
Error in rule contigs_sig:
    jobid: 25
    output: output.demo/stage1/GCA_001593925.1_ASM159392v1_genomic.fna.gz.sig
    conda-env: /path/to/tmp/tree/charcoal/.snakemake/conda/16fae472
    shell:

        sourmash compute -k 31 --scaled 1000             --dna demo/genomes/GCA_001593925.1_ASM159392v1_genomic.fna.gz -o output.demo/stage1/GCA_001593925.1_ASM159392v1_genomic.fna.gz.sig

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/tmp/tree/charcoal/.snakemake/log/2021-01-11T164858.006949.snakemake.log
Error in snakemake invocation: Command '['snakemake', '-s', '/path/to/tmp/tree/charcoal/charcoal/Snakefile', '--use-conda', '-j', '1', '-j', '4', '--configfile', '/path/to/tmp/tree/charcoal/charcoal/conf/defaults.conf', '/path/to/tmp/tree/charcoal/charcoal/conf/system.conf', 'demo/demo.conf']' returned non-zero exit status 1.

Ah, nevermind, sourmash is missing, easy fix? No:

python -m charcoal run demo/demo.conf -j 4

Traceback (most recent call last):
  File "/path/to/miniconda3/envs/charcoal/bin/snakemake", line 6, in <module>
    from snakemake import main
  File "/path/to/miniconda3/envs/charcoal/site-packages/snakemake/__init__.py", line 20, in <module>
    from snakemake.workflow import Workflow
  File "/path/to/miniconda3/envs/charcoal/site-packages/snakemake/workflow.py", line 30, in <module>
    from snakemake.shell import shell
  File "/path/to/miniconda3/envs/charcoal/site-packages/snakemake/shell.py", line 16, in <module>
    from snakemake import singularity
  File "/path/to/miniconda3/envs/charcoal/site-packages/snakemake/singularity.py", line 9, in <module>
    from snakemake.conda import Conda
  File "/path/to/miniconda3/envs/charcoal/site-packages/snakemake/conda.py", line 5, in <module>
    from urllib.request import urlopen
  File "/path/to/miniconda3/envs/charcoal/lib-python/3/urllib/request.py", line 2616, in <module>
    from _scproxy import _get_proxy_settings, _get_proxies
  File "/path/to/miniconda3/envs/charcoal/lib_pypy/_scproxy.py", line 64, in <module>
    ffi = _CFSetup()
  File "/path/to/miniconda3/envs/charcoal/lib_pypy/_scproxy.py", line 63, in _CFSetup
    return CFProxy()
  File "/path/to/miniconda3/envs/charcoal/lib_pypy/_scproxy.py", line 52, in __init__
    func = getattr(mod, fname)
  File "/path/to/miniconda3/envs/charcoal/lib-python/3/ctypes/__init__.py", line 366, in __getattr__
    func = self.__getitem__(name)
  File "/path/to/miniconda3/envs/charcoal/lib-python/3/ctypes/__init__.py", line 371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
  File "/path/to/miniconda3/envs/charcoal/lib_pypy/_ctypes/function.py", line 255, in __init__
    ptr = self._getfuncptr([], ctypes.c_int)
  File "/path/to/miniconda3/envs/charcoal/lib_pypy/_ctypes/function.py", line 442, in _getfuncptr
    self._ptr = cdll.getfunc(self.name, ffi_argtypes, ffi_restype)
AttributeError: No symbol SCDynamicStoreCopyProxies found in library <None>
Error in snakemake invocation: Command '['snakemake', '-s', '/path/to/tmp/tree/charcoal/charcoal/Snakefile', '--use-conda', '-j', '1', '-j', '4', '--configfile', '/path/to/tmp/tree/charcoal/charcoal/conf/defaults.conf', '/path/to/tmp/tree/charcoal/charcoal/conf/system.conf', 'demo/demo.conf']' returned non-zero exit status 1.

Not being much of a snakemake man myself, I am a bit at a loss to what SCDynamicStoreCopyProxies means, and so is Google. Any idea what goes wrong here?

Thanks!

[taylorreiter]

Was this on an M1 mac?

[phiweger]

No its a Quad-Core Intel Core i7.

[taylorreiter]

thanks @phiweger, i'm slowly making my way through the issues on the demo, i'll try and track this down

[taylorreiter]

this is no longer occurring with the current envs/sourmash version on my mac computer...I'm going to let this rest for now, but leave it open so it's more easily findable. thanks for reporting, and sorry for the huge delay.