search

dohlee / chromoformer

The official code implementation for Chromoformer in PyTorch. (Lee et al., Nature Communications. 2022)
GNU General Public License v3.0
31 stars 2 forks source link

Issue running data processing #3

Closed Al-Murphy closed 1 year ago

Al-Murphy commented 1 year ago

Congrats on a really great model!

I'm trying to run the snakemake file to install and process the necessary ROADMAP (and pcHi-C) data. However I get the following error:

(chromoformer) snakemake --resources network=5 -j 32 --restart-times 3 --use-conda --conda-frontend mamba -pr
Building DAG of jobs...
Creating conda environment environment.yaml...
Downloading and installing remote packages.
Environment for environment.yaml created (location: .snakemake/conda/819bfa1bb63d054f8b22a32f4f42dee5)
Using shell: /usr/bin/bash
Provided cores: 32
Rules claiming more threads will be scaled down.
Provided resources: network=5
Traceback (most recent call last):
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/snakemake/__init__.py", line 696, in snakemake
    success = workflow.execute(
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/snakemake/workflow.py", line 1037, in execute
    logger.run_info("\n".join(dag.stats()))
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/snakemake/dag.py", line 2159, in stats
    yield tabulate(rows, headers="keys")
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/tabulate/__init__.py", line 2048, in tabulate
    list_of_lists, headers = _normalize_tabular_data(
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/tabulate/__init__.py", line 1471, in _normalize_tabular_data
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/tabulate/__init__.py", line 1471, in <lambda>
    rows = list(map(lambda r: r if _is_separating_line(r) else list(r), rows))
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/tabulate/__init__.py", line 107, in _is_separating_line
    (len(row) >= 1 and row[0] == SEPARATING_LINE)
  File "/rds/general/user/aemurphy/home/anaconda3/envs/chromoformer/lib/python3.9/site-packages/snakemake/rules.py", line 1120, in __eq__
    return self.name == other.name and self.output == other.output
AttributeError: 'str' object has no attribute 'name'

I guess this is likely a version issue, I have python v3.9.15 and snakemake v6.5.3. Full list of versions are below:

Package versions Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge aioeasywebdav 2.4.0 py39hf3d152e_1001 conda-forge aiohttp 3.8.3 py39hb9d737c_1 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge amply 0.1.5 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge atk-1.0 2.38.0 hd4edc92_1 conda-forge attmap 0.13.2 pyhd8ed1ab_0 conda-forge attrs 22.1.0 pyh71513ae_1 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bcrypt 3.2.2 py39hb9d737c_1 conda-forge bedtools 2.23.0 h5b5514e_6 bioconda boto3 1.26.21 pyhd8ed1ab_0 conda-forge botocore 1.29.21 pyhd8ed1ab_0 conda-forge brotlipy 0.7.0 py39hb9d737c_1005 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2022.9.24 ha878542_0 conda-forge cachetools 5.2.0 pyhd8ed1ab_0 conda-forge cairo 1.16.0 ha61ee94_1014 conda-forge certifi 2022.9.24 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py39he91dace_2 conda-forge charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge coin-or-cbc 2.10.8 h3786ebc_0 conda-forge coin-or-cgl 0.60.6 h6f57e76_2 conda-forge coin-or-clp 1.17.7 hc56784d_2 conda-forge coin-or-osi 0.108.7 h2720bb7_2 conda-forge coin-or-utils 2.11.6 h202d8b1_2 conda-forge coincbc 2.10.8 0_metapackage conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge commonmark 0.9.1 py_0 conda-forge configargparse 1.5.3 pyhd8ed1ab_0 conda-forge connection_pool 0.0.3 pyhd3deb0d_0 conda-forge cryptography 38.0.4 py39h3ccb8fc_0 conda-forge cudatoolkit 11.1.1 ha002fc5_11 conda-forge cudnn 8.4.1.50 hed8a83a_0 conda-forge dataclasses 0.8 pyhc8e2a94_3 conda-forge datrie 0.8.2 py39hb9d737c_6 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge docutils 0.19 py39hf3d152e_1 conda-forge dropbox 11.36.0 pyhd8ed1ab_0 conda-forge exceptiongroup 1.0.4 pyhd8ed1ab_0 conda-forge expat 2.5.0 h27087fc_0 conda-forge fftw 3.3.10 nompi_hf0379b8_106 conda-forge filechunkio 1.8 py_2 conda-forge filelock 3.8.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.1 hc2a2eb6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge freetype 2.12.1 hca18f0e_1 conda-forge fribidi 1.0.10 h36c2ea0_0 conda-forge frozenlist 1.3.3 py39hb9d737c_0 conda-forge ftputil 5.0.4 pyhd8ed1ab_0 conda-forge future 0.18.2 pyhd8ed1ab_6 conda-forge gdk-pixbuf 2.42.8 hff1cb4f_1 conda-forge gettext 0.21.1 h27087fc_0 conda-forge ghostscript 9.54.0 h27087fc_2 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge gitdb 4.0.10 pyhd8ed1ab_0 conda-forge gitpython 3.1.29 pyhd8ed1ab_0 conda-forge google-api-core 2.10.0 pyhd8ed1ab_0 conda-forge google-api-python-client 2.68.0 pyhd8ed1ab_0 conda-forge google-auth 2.15.0 pyh1a96a4e_0 conda-forge google-auth-httplib2 0.1.0 pyhd8ed1ab_1 conda-forge google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge google-cloud-storage 2.6.0 pyh1a96a4e_0 conda-forge google-crc32c 1.1.2 py39h8db88ab_4 conda-forge google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge googleapis-common-protos 1.57.0 py39hf3d152e_0 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge graphviz 6.0.2 h99bc08f_0 conda-forge grpcio 1.46.3 py39h7fbbf82_0 conda-forge gtk2 2.24.33 h90689f9_2 conda-forge gts 0.7.6 h64030ff_2 conda-forge harfbuzz 5.3.0 h418a68e_0 conda-forge httplib2 0.21.0 pyhd8ed1ab_0 conda-forge icu 70.1 h27087fc_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge imagemagick 7.1.0_52 pl5321hc735599_0 conda-forge importlib-metadata 5.1.0 pyha770c72_0 conda-forge importlib_resources 5.10.0 pyhd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge intel-openmp 2021.4.0 h06a4308_3561 jbig 2.1 h7f98852_2003 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jmespath 1.0.1 pyhd8ed1ab_0 conda-forge joblib 1.2.0 pyhd8ed1ab_0 conda-forge jpeg 9e h166bdaf_2 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge jupyter_core 5.1.0 py39hf3d152e_0 conda-forge lcms2 2.14 h6ed2654_0 conda-forge ld_impl_linux-64 2.39 hcc3a1bd_1 conda-forge ldc 1.13.0 h02c9852_1 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20220623.0 cxx17_h48a1fff_5 conda-forge libblas 3.9.0 12_linux64_mkl conda-forge libcblas 3.9.0 12_linux64_mkl conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libdeflate 1.14 h166bdaf_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 12.2.0 h65d4601_19 conda-forge libgd 2.3.3 h18fbbfe_3 conda-forge libgfortran-ng 12.2.0 h69a702a_19 conda-forge libgfortran5 12.2.0 h337968e_19 conda-forge libglib 2.74.1 h606061b_1 conda-forge libgomp 12.2.0 h65d4601_19 conda-forge libiconv 1.17 h166bdaf_0 conda-forge liblapack 3.9.0 12_linux64_mkl conda-forge liblapacke 3.9.0 12_linux64_mkl conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge libpng 1.6.39 h753d276_0 conda-forge libprotobuf 3.16.0 h780b84a_0 conda-forge librsvg 2.54.4 h7abd40a_0 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsqlite 3.40.0 h753d276_0 conda-forge libstdcxx-ng 12.2.0 h46fd767_19 conda-forge libtiff 4.4.0 h55922b4_4 conda-forge libtool 2.4.6 h9c3ff4c_1008 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp 1.2.4 h522a892_0 conda-forge libwebp-base 1.2.4 h166bdaf_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.10.3 h7463322_0 conda-forge libzlib 1.2.13 h166bdaf_4 conda-forge logmuse 0.2.6 pyh8c360ce_0 conda-forge magma 2.5.4 h5fc5d34_4 conda-forge markupsafe 2.1.1 py39hb9d737c_2 conda-forge mkl 2021.4.0 h06a4308_640 multidict 6.0.2 py39hb9d737c_2 conda-forge nbformat 5.7.0 pyhd8ed1ab_0 conda-forge nccl 2.14.3.1 h12f7317_0 conda-forge ncurses 6.3 h27087fc_1 conda-forge networkx 2.8.8 pyhd8ed1ab_0 conda-forge ninja 1.11.0 h924138e_0 conda-forge numpy 1.21.0 py39hdbf815f_0 conda-forge oauth2client 4.1.3 py_0 conda-forge openjpeg 2.5.0 h7d73246_1 conda-forge openssl 3.0.7 h166bdaf_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.3.0 py39hde0f152_0 conda-forge pango 1.50.12 h382ae3d_0 conda-forge paramiko 2.12.0 pyhd8ed1ab_0 conda-forge pcre2 10.40 hc3806b6_0 conda-forge peppy 0.35.3 pyhd8ed1ab_0 conda-forge perl 5.32.1 2_h7f98852_perl5 conda-forge pillow 9.2.0 py39hf3a2cdf_3 conda-forge pip 22.3.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pkg-config 0.29.2 h36c2ea0_1008 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 2.5.2 pyhd8ed1ab_1 conda-forge pluggy 1.0.0 pyhd8ed1ab_5 conda-forge ply 3.11 py_1 conda-forge prettytable 3.5.0 pyhd8ed1ab_0 conda-forge protobuf 3.16.0 py39he80948d_0 conda-forge psutil 5.9.4 py39hb9d737c_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge pulp 2.7.0 py39hf3d152e_0 conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.13.0 pyhd8ed1ab_0 conda-forge pygraphviz 1.10 py39h8e0b5b3_2 conda-forge pynacl 1.5.0 py39hb9d737c_2 conda-forge pyopenssl 22.1.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyrsistent 0.19.2 py39hb9d737c_0 conda-forge pysftp 0.2.9 py_1 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge pytest 7.2.0 pyhd8ed1ab_2 conda-forge python 3.9.15 hba424b6_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge python-irodsclient 1.1.5 pyhd8ed1ab_0 conda-forge python_abi 3.9 3_cp39 conda-forge pytorch 1.9.0 cuda111py39hc274426_1 conda-forge pytorch-gpu 1.9.0 cuda111py39h788eb59_1 conda-forge pytz 2022.6 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py39hb9d737c_5 conda-forge ratelimiter 1.2.0 pyhd8ed1ab_1003 conda-forge re2 2022.06.01 h27087fc_1 conda-forge readline 8.1.2 h0f457ee_0 conda-forge requests 2.28.1 pyhd8ed1ab_1 conda-forge rich 12.6.0 pyhd8ed1ab_0 conda-forge rsa 4.9 pyhd8ed1ab_0 conda-forge s3transfer 0.6.0 pyhd8ed1ab_0 conda-forge sambamba 0.6.8 h89e63da_2 bioconda scikit-learn 0.24.2 py39h7c5d8c9_1 conda-forge scipy 1.9.3 py39hddc5342_2 conda-forge setuptools 59.8.0 py39hf3d152e_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge slacker 0.14.0 py_0 conda-forge sleef 3.5.1 h9b69904_2 conda-forge smart_open 6.2.0 pyha770c72_0 conda-forge smmap 3.0.5 pyh44b312d_0 conda-forge snakemake 6.5.3 hdfd78af_0 bioconda snakemake-minimal 6.5.3 pyhdfd78af_0 bioconda stone 3.3.1 pyhd8ed1ab_0 conda-forge stopit 1.1.2 py_0 conda-forge tabulate 0.9.0 pyhd8ed1ab_1 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toposort 1.7 pyhd8ed1ab_0 conda-forge torchaudio 0.9.0 py39 pytorch torchvision 0.10.0 py39cuda111hcd06603_0_cuda conda-forge tqdm 4.61.2 pyhd8ed1ab_1 conda-forge traitlets 5.6.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.4.0 hd8ed1ab_0 conda-forge typing_extensions 4.4.0 pyha770c72_0 conda-forge tzdata 2022g h191b570_0 conda-forge ubiquerg 0.6.2 pyhd8ed1ab_0 conda-forge uritemplate 4.1.1 pyhd8ed1ab_0 conda-forge urllib3 1.26.13 pyhd8ed1ab_0 conda-forge veracitools 0.1.3 py_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge wheel 0.38.4 pyhd8ed1ab_0 conda-forge wrapt 1.14.1 py39hb9d737c_1 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-libxt 1.2.1 h7f98852_2 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.8.1 py39hb9d737c_0 conda-forge zipp 3.11.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h166bdaf_4 conda-forge zstd 1.5.2 h6239696_4 conda-forge <\details> Thanks, Alan
dohlee commented 1 year ago

Thanks for specifying the package versions with the issue. I also suspect some kind of version issue regarding snakemake, and I'll try to reproduce the error to clarify it.

I'll let you know if I find something!

dohlee commented 1 year ago

I'm still working on it, but fortunately I found through google that it's due to the incompatibility issue between tabulate 0.9 and an older version of snakemake (<7.15.2), which is resolved by https://github.com/bioconda/bioconda-recipes/pull/37359. I see that you have tabulate 0.9.

So maybe updating snakemake to more recent version or downgrading tabulate to 0.8 may solve the issue. You may give it a try! Nevertheless, I'll keep trying to reproduce your error.

Thanks, Dohoon

dohlee commented 1 year ago

Finally, I could reproduce your error with the package versions you specified.

error

and updating snakemake to the latest version (7.8.12) solves the error: mamba update -c bioconda snakemake=7.18.2

image

Hope this helps! Dohoon

Al-Murphy commented 1 year ago

Updating Snakemake (to version 7.18.2) worked! Thank you very much for the help Dohoon!