search

loosolab / denis

framework for de novo motif generation on ATAC-seq
MIT License
2 stars 0 forks source link

Snakemake v8.25.0 error #1

Open hschult opened 3 weeks ago

hschult commented 3 weeks ago

DENIS hits an error with snakemake v8.25.0:

rule all:
    input:
        output_files
    message: f"Pipeline done. Started {start_time}"    

^ Seems to be from line 54 of the Snakefile.

Downgrading snakemake to v7.3.7 resolves the error.

CebolaLab commented 3 weeks ago

Workflow exits with the following error:

Running on a PBS system:

Building DAG of jobs...
Creating conda environment environments/bigwig.yml...
Downloading and installing remote packages.
Environment for /xxxxxxxxx/denis/snakefiles/../environments/bigwig.yml created (location: .snakemake/conda/8b761192e7c474111471a21e65fd14b7)
Creating conda environment environments/motif.yml...
Downloading and installing remote packages.
Environment for /xxxxxxxx/denis/snakefiles/../environments/motif.yml created (location: .snakemake/conda/024f814690ea462e2d5954805d1047d6)
Using shell: /bin/bash
Provided cores: 64
Rules claiming more threads will be scaled down.
Traceback (most recent call last):
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/snakemake/__init__.py", line 722, in snakemake
    success = workflow.execute(
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/snakemake/workflow.py", line 1092, in execute
    logger.run_info("\n".join(dag.stats()))
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/snakemake/dag.py", line 2248, in stats
    yield tabulate(rows, headers="keys")
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/tabulate/__init__.py", line 2048, in tabulate
    list_of_lists, headers = _normalize_tabular_data(
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/tabulate/__init__.py", line 1471, in _normalize_tabular_data
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/xxxxxxxx/envs/snakemake-denis/lib/python3.10/site-packages/tabulate/__init__.py", line 1471, in <lambda>
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/tabulate/__init__.py", line 107, in _is_separating_line
    (len(row) >= 1 and row[0] == SEPARATING_LINE)
  File "/xxxxxxxx/anaconda3/envs/snakemake-denis/lib/python3.10/site-packages/snakemake/rules.py", line 1213, in __eq__
    return self.name == other.name and self.output == other.output
AttributeError: 'str' object has no attribute 'name'

Using conda environment:

name: snakemake-denis
channels:
  - conda-forge
  - mathieu
  - bioconda
  - r
  - imperial-college-research-computing
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - aioeasywebdav=2.4.0=pyha770c72_0
  - aiohappyeyeballs=2.4.3=pyhd8ed1ab_0
  - aiohttp=3.10.10=py310h89163eb_0
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - amply=0.1.6=pyhd8ed1ab_0
  - annotated-types=0.7.0=pyhd8ed1ab_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - async-timeout=4.0.3=pyhd8ed1ab_0
  - attrs=24.2.0=pyh71513ae_0
  - bcrypt=4.2.0=py310h505e2c1_1
  - boto3=1.35.52=pyhd8ed1ab_0
  - botocore=1.35.53=pyge310_1234567_0
  - brotli-python=1.1.0=py310hf71b8c6_2
  - bzip2=1.0.8=h4bc722e_7
  - c-ares=1.34.2=heb4867d_0
  - ca-certificates=2024.8.30=hbcca054_0
  - cachetools=5.5.0=pyhd8ed1ab_0
  - certifi=2024.8.30=pyhd8ed1ab_0
  - cffi=1.17.1=py310h8deb56e_0
  - charset-normalizer=3.4.0=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - coin-or-cbc=2.10.12=h8b142ea_1
  - coin-or-cgl=0.60.9=h1d3f3f2_0
  - coin-or-clp=1.17.10=h07f2a63_0
  - coin-or-osi=0.108.11=h6514dde_1
  - coin-or-utils=2.11.12=h99da652_1
  - coincbc=2.10.12=1_metapackage
  - colorama=0.4.6=pyhd8ed1ab_0
  - coloredlogs=15.0.1=pyhd8ed1ab_3
  - configargparse=1.7=pyhd8ed1ab_0
  - connection_pool=0.0.3=pyhd3deb0d_0
  - cryptography=43.0.3=py310h6c63255_0
  - datrie=0.8.2=py310ha75aee5_8
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - docutils=0.21.2=pyhd8ed1ab_0
  - dpath=2.2.0=pyha770c72_0
  - dropbox=11.36.2=pyhd8ed1ab_0
  - exceptiongroup=1.2.2=pyhd8ed1ab_0
  - filechunkio=1.8=py_2
  - filelock=3.16.1=pyhd8ed1ab_0
  - frozenlist=1.5.0=py310ha75aee5_0
  - ftputil=5.1.0=pyhd8ed1ab_0
  - gitdb=4.0.11=pyhd8ed1ab_0
  - gitpython=3.1.43=pyhd8ed1ab_0
  - google-api-core=2.22.0=pyhd8ed1ab_0
  - google-api-python-client=2.151.0=pyhff2d567_0
  - google-auth=2.35.0=pyhff2d567_0
  - google-auth-httplib2=0.2.0=pyhd8ed1ab_0
  - google-cloud-core=2.4.1=pyhd8ed1ab_0
  - google-cloud-storage=2.18.2=pyhff2d567_0
  - google-crc32c=1.1.2=py310hd027165_6
  - google-resumable-media=2.7.2=pyhd8ed1ab_1
  - googleapis-common-protos=1.65.0=pyhd8ed1ab_0
  - grpcio=1.67.1=py310h1a6248f_0
  - h2=4.1.0=pyhd8ed1ab_0
  - hpack=4.0.0=pyh9f0ad1d_0
  - httplib2=0.22.0=pyhd8ed1ab_0
  - humanfriendly=10.0=pyhd8ed1ab_6
  - hyperframe=6.0.1=pyhd8ed1ab_0
  - idna=3.10=pyhd8ed1ab_0
  - importlib_resources=6.4.5=pyhd8ed1ab_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - jinja2=3.1.4=pyhd8ed1ab_0
  - jmespath=1.0.1=pyhd8ed1ab_0
  - jsonschema=4.23.0=pyhd8ed1ab_0
  - jsonschema-specifications=2024.10.1=pyhd8ed1ab_0
  - jupyter_core=5.7.2=pyh31011fe_1
  - ld_impl_linux-64=2.43=h712a8e2_2
  - libabseil=20240722.0=cxx17_h5888daf_1
  - libblas=3.9.0=25_linux64_openblas
  - libcblas=3.9.0=25_linux64_openblas
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libffi=3.4.2=h7f98852_5
  - libgcc=14.2.0=h77fa898_1
  - libgcc-ng=14.2.0=h69a702a_1
  - libgfortran=14.2.0=h69a702a_1
  - libgfortran-ng=14.2.0=h69a702a_1
  - libgfortran5=14.2.0=hd5240d6_1
  - libgomp=14.2.0=h77fa898_1
  - libgrpc=1.67.1=hc2c308b_0
  - liblapack=3.9.0=25_linux64_openblas
  - liblapacke=3.9.0=25_linux64_openblas
  - libnsl=2.0.1=hd590300_0
  - libopenblas=0.3.28=pthreads_h94d23a6_0
  - libprotobuf=5.28.2=h5b01275_0
  - libre2-11=2024.07.02=hbbce691_1
  - libsodium=1.0.20=h4ab18f5_0
  - libsqlite=3.47.0=hadc24fc_1
  - libstdcxx=14.2.0=hc0a3c3a_1
  - libstdcxx-ng=14.2.0=h4852527_1
  - libuuid=2.38.1=h0b41bf4_0
  - libxcrypt=4.4.36=hd590300_1
  - libzlib=1.3.1=hb9d3cd8_2
  - logmuse=0.2.8=pyhd8ed1ab_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=3.0.2=py310h89163eb_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - multidict=6.1.0=py310h89163eb_1
  - nbformat=5.10.4=pyhd8ed1ab_0
  - ncurses=6.5=he02047a_1
  - numpy=2.1.2=py310hd6e36ab_0
  - oauth2client=4.1.3=pyhd8ed1ab_1
  - openssl=3.3.2=hb9d3cd8_0
  - packaging=24.1=pyhd8ed1ab_0
  - pandas=2.2.3=py310h5eaa309_1
  - paramiko=3.5.0=pyhd8ed1ab_0
  - pephubclient=0.4.4=pyhd8ed1ab_0
  - peppy=0.40.7=pyhd8ed1ab_1
  - pip=24.3.1=pyh8b19718_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_1
  - plac=1.4.3=pyhd8ed1ab_0
  - platformdirs=4.3.6=pyhd8ed1ab_0
  - pluggy=1.5.0=pyhd8ed1ab_0
  - ply=3.11=pyhd8ed1ab_2
  - prettytable=3.12.0=pyhd8ed1ab_0
  - propcache=0.2.0=py310ha75aee5_2
  - proto-plus=1.25.0=pyhd8ed1ab_0
  - protobuf=5.28.2=py310hf71b8c6_0
  - psutil=6.1.0=py310ha75aee5_0
  - pulp=2.7.0=py310hff52083_1
  - py=1.11.0=pyh6c4a22f_0
  - pyasn1=0.6.1=pyhd8ed1ab_1
  - pyasn1-modules=0.4.1=pyhd8ed1ab_0
  - pycparser=2.22=pyhd8ed1ab_0
  - pydantic=2.9.2=pyhd8ed1ab_0
  - pydantic-core=2.23.4=py310h505e2c1_0
  - pygments=2.18.0=pyhd8ed1ab_0
  - pynacl=1.5.0=py310ha75aee5_4
  - pyopenssl=24.2.1=pyhd8ed1ab_2
  - pyparsing=3.2.0=pyhd8ed1ab_1
  - pysftp=0.2.9=py_1
  - pysocks=1.7.1=pyha2e5f31_6
  - pytest=8.3.3=pyhd8ed1ab_0
  - python=3.10.15=h4a871b0_2_cpython
  - python-dateutil=2.9.0=pyhd8ed1ab_0
  - python-fastjsonschema=2.20.0=pyhd8ed1ab_0
  - python-irodsclient=2.1.0=pyhd8ed1ab_0
  - python-tzdata=2024.2=pyhd8ed1ab_0
  - python_abi=3.10=5_cp310
  - pytz=2024.1=pyhd8ed1ab_0
  - pyu2f=0.1.5=pyhd8ed1ab_0
  - pyyaml=6.0.2=py310ha75aee5_1
  - ratelimiter=1.2.0=pyhd8ed1ab_1003
  - re2=2024.07.02=h77b4e00_1
  - readline=8.2=h8228510_1
  - referencing=0.35.1=pyhd8ed1ab_0
  - requests=2.32.3=pyhd8ed1ab_0
  - retry=0.9.2=py_0
  - rich=13.9.3=pyhd8ed1ab_0
  - rpds-py=0.20.0=py310h505e2c1_1
  - rsa=4.9=pyhd8ed1ab_0
  - s3transfer=0.10.3=pyhd8ed1ab_0
  - setuptools=75.3.0=pyhd8ed1ab_0
  - setuptools-scm=8.1.0=pyhd8ed1ab_0
  - shellingham=1.5.4=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - slacker=0.14.0=py_0
  - smart_open=7.0.5=pyhd8ed1ab_1
  - smmap=5.0.0=pyhd8ed1ab_0
  - snakemake=7.3.7=hdfd78af_0
  - snakemake-minimal=7.3.7=pyhdfd78af_0
  - stone=3.3.8=pyhd8ed1ab_0
  - stopit=1.1.2=py_0
  - tabulate=0.9.0=pyhd8ed1ab_1
  - tk=8.6.13=noxft_h4845f30_101
  - tomli=2.0.2=pyhd8ed1ab_0
  - toposort=1.10=pyhd8ed1ab_0
  - traitlets=5.14.3=pyhd8ed1ab_0
  - typer=0.12.5=pyhd8ed1ab_0
  - typer-slim=0.12.5=pyhd8ed1ab_0
  - typer-slim-standard=0.12.5=hd8ed1ab_0
  - typing-extensions=4.12.2=hd8ed1ab_0
  - typing_extensions=4.12.2=pyha770c72_0
  - tzdata=2024b=hc8b5060_0
  - ubiquerg=0.8.0=pyhd8ed1ab_0
  - uritemplate=4.1.1=pyhd8ed1ab_0
  - urllib3=2.2.3=pyhd8ed1ab_0
  - veracitools=0.1.3=py_0
  - wcwidth=0.2.13=pyhd8ed1ab_0
  - wheel=0.44.0=pyhd8ed1ab_0
  - wrapt=1.16.0=py310ha75aee5_1
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yarl=1.16.0=py310ha75aee5_0
  - yte=1.5.4=pyha770c72_0
  - zipp=3.20.2=pyhd8ed1ab_0
  - zstandard=0.23.0=py310ha39cb0e_1
  - zstd=1.5.6=ha6fb4c9_0
prefix: /xxxxxxxx/anaconda3/envs/snakemake-denis
hschult commented 2 weeks ago

Please excuse the wait time, this turned out to be three different issues at once, which resulted in a lot of trial and error on my side.

The first error did occur because of a missing new line at the end of my Snakefile. Older versions of Snakemake were fine with this but newer versions (e.g. Snakemake 8.25.0) are not. This is fixed and already on the main branch (#2). The second error, which you reported for Snakemake 7.3.7 is a known bug caused by a dependency (see here). It can be manually fixed by installing tabulate=0.8.10 but is also fixed in Snakemake >=7.15.2. Upgrading Snakemake to 8.25.0 led me to the third bug, which is discussed here. From what I can tell, this was introduced with Snakemake 8 and unfortunately does not have a solution, yet. My recommendation is to use Snakemake >=7.15.2,<8.

Tldr: This pipeline is broken in Snakemake >=8. Use one installed with e.g. this: mamba install "snakemake>=7.15.2,<8".