josephhughes commented 4 years ago

Hi @aineniamh,

I tried to run it from our server and got the following error:

sent 25 bytes  received 320 bytes  46.00 bytes/sec
total size is 2.47G  speedup is 7,159,035.33
Found cog data:
    - /home4/nCov/Joseph/civet/civet-cat/cog_alignment.fasta
    - /home4/nCov/Joseph/civet/civet-cat/cog_global_alignment.fasta
    - /home4/nCov/Joseph/civet/civet-cat/cog_metadata.csv
    - /home4/nCov/Joseph/civet/civet-cat/cog_metadata_all.csv
    - /home4/nCov/Joseph/civet/civet-cat/cog_global_metadata.csv
    - /home4/nCov/Joseph/civet/civet-cat/cog_global_tree.nexus
The reference genome is found /home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/data/reference.fasta
The font file is /home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/data/HelveticaNeue.ttf
Found the report_template /home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/scripts/civet_template.pmd
AttributeError in line 157 of /home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/scripts/assess_input_file.smk:
'Workflow' object has no attribute 'cores'
  File "/home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/scripts/Snakefile", line 33, in <module>
  File "/home1/hugh01j/miniconda3/envs/civet/lib/python3.6/site-packages/civet-0.1-py3.6.egg/civet/scripts/assess_input_file.smk", line 157, in <module>

aineniamh commented 4 years ago

Ah, I've seen this error before. Can you check what version of snakemake you're running? We have in the environment file snakemake-minimal=5.13, I think earlier versions didn't have --cores

josephhughes commented 4 years ago

I've got snakemake 5.19.3 but do I actually need snakemake-minimal?

aineniamh commented 4 years ago

Regular snakemake should work as well and with that version it should have the cores attribute (docs here). If you're running from within the civet environment, which your code suggests you are, there may be some version conflict going on.

If you double check what version is running within that environment specifically and potentially update if needed. 'Workflow' object has no attribute 'cores' is coming direct from snakemake.

josephhughes commented 4 years ago

When I do: conda list in the civet conda environment, I get: `# packages in environment at /home1/hugh01j/miniconda3/envs/civet: #

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_llvm conda-forge aioeasywebdav 2.4.0 py36_1000 conda-forge aiohttp 3.6.2 py36h516909a_0 conda-forge appdirs 1.4.3 py_1 conda-forge async-generator 1.10 pypi_0 pypi async-timeout 3.0.1 py_1000 conda-forge attrs 19.3.0 py_0 conda-forge backcall 0.2.0 pypi_0 pypi bcrypt 3.1.7 py36h8c4c3a4_1 conda-forge biopython 1.74 py36h516909a_0 conda-forge bleach 3.1.5 pypi_0 pypi boto3 1.14.3 pypi_0 pypi botocore 1.17.3 pypi_0 pypi brotlipy 0.7.0 py36h8c4c3a4_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge ca-certificates 2020.4.5.2 hecda079_0 conda-forge cached-property 1.5.1 pypi_0 pypi cachetools 4.1.0 py_1 conda-forge cairo 1.16.0 h3fc0475_1004 conda-forge certifi 2020.4.5.2 py36h9f0ad1d_0 conda-forge cffi 1.14.0 py36hd463f26_0 conda-forge chardet 3.0.4 py36h9f0ad1d_1006 conda-forge civet 0.1 pypi_0 pypi click 7.1.2 pypi_0 pypi clusterfunk 0.0.2 pypi_0 pypi conda 4.8.3 py36h9f0ad1d_1 conda-forge conda-package-handling 1.6.0 py36h8c4c3a4_2 conda-forge configargparse 1.2.3 pyh9f0ad1d_0 conda-forge crc32c 2.0 py36h516909a_0 conda-forge cryptography 2.9.2 py36h45558ae_0 conda-forge cycler 0.10.0 pypi_0 pypi datafunk 0.0.7 pypi_0 pypi datapackage 1.14.1 pypi_0 pypi datrie 0.8.2 py36h8c4c3a4_0 conda-forge decorator 4.4.2 py_0 conda-forge defusedxml 0.7.0rc1 pypi_0 pypi docutils 0.16 py36h9f0ad1d_1 conda-forge dropbox 10.1.1 pyh9f0ad1d_0 conda-forge entrypoints 0.3 pypi_0 pypi epiweeks 2.1.2 pypi_0 pypi et-xmlfile 1.0.1 pypi_0 pypi expat 2.2.9 he1b5a44_2 conda-forge fftw 3.3.8 mpi_mpich_h3f9e1be_1010 conda-forge filechunkio 1.8 py_2 conda-forge fontconfig 2.13.1 h1056068_1002 conda-forge freetype 2.10.2 he06d7ca_0 conda-forge ftputil 4.0.0 py_0 conda-forge gdk-pixbuf 2.36.12 h3f25603_1005 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge ghostscript 9.22 hf484d3e_1001 conda-forge giflib 5.1.7 h516909a_1 conda-forge gitdb 4.0.5 py_0 conda-forge gitpython 3.1.3 py_0 conda-forge glib 2.64.3 h6f030ca_0 conda-forge gobject-introspection 1.64.1 py36h6d43c07_1 conda-forge google-api-core 1.20.0 pypi_0 pypi google-api-python-client 1.9.3 pyh9f0ad1d_0 conda-forge google-auth 1.16.1 pypi_0 pypi google-auth-httplib2 0.0.3 py_3 conda-forge google-cloud-core 1.3.0 py_0 conda-forge google-cloud-storage 1.28.1 pyh9f0ad1d_0 conda-forge google-resumable-media 0.5.1 pyh9f0ad1d_0 conda-forge googleapis-common-protos 1.52.0 pypi_0 pypi graphite2 1.3.13 he1b5a44_1001 conda-forge graphviz 2.38.0 hf68f40c_1011 conda-forge harfbuzz 2.4.0 hee91db6_5 conda-forge httplib2 0.18.1 pyh9f0ad1d_0 conda-forge icu 67.1 he1b5a44_0 conda-forge idna 2.9 py_1 conda-forge idna_ssl 1.1.0 py36_1000 conda-forge ijson 3.0.4 pypi_0 pypi imagemagick 7.0.8_54 pl526h39023e4_0 conda-forge importlib-metadata 1.6.1 py36h9f0ad1d_0 conda-forge importlib_metadata 1.6.1 0 conda-forge ipykernel 5.3.0 pypi_0 pypi ipython 7.15.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipython_genutils 0.2.0 py_1 conda-forge iqtree 1.6.12 he513fc3_1 bioconda isodate 0.6.0 pypi_0 pypi jbig 2.1 h516909a_2002 conda-forge jdcal 1.4.1 pypi_0 pypi jedi 0.17.0 pypi_0 pypi jinja2 3.0.0a1 pypi_0 pypi jmespath 0.10.0 pyh9f0ad1d_0 conda-forge jpeg 9d h516909a_0 conda-forge jsonlines 1.2.0 pypi_0 pypi jsonpointer 2.0 pypi_0 pypi jsonschema 3.2.0 py36h9f0ad1d_1 conda-forge jupyter-client 6.1.3 pypi_0 pypi jupyter_core 4.6.3 py36h9f0ad1d_1 conda-forge jupyterlab-pygments 0.1.1 pypi_0 pypi kiwisolver 1.2.0 pypi_0 pypi krb5 1.17.1 h2fd8d38_0 conda-forge ld_impl_linux-64 2.34 h53a641e_5 conda-forge libarchive 3.3.3 h3a8160c_1008 conda-forge libblas 3.8.0 16_openblas conda-forge libcblas 3.8.0 16_openblas conda-forge libcroco 0.6.13 h8d621e5_1 conda-forge libcurl 7.69.1 hf7181ac_0 conda-forge libedit 3.1.20191231 h46ee950_0 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgfortran-ng 7.5.0 hdf63c60_6 conda-forge libiconv 1.15 h516909a_1006 conda-forge liblapack 3.8.0 16_openblas conda-forge libopenblas 0.3.9 h5ec1e0e_0 conda-forge libpng 1.6.37 hed695b0_1 conda-forge libprotobuf 3.12.3 h8b12597_0 conda-forge librsvg 2.44.14 h11c8777_0 conda-forge libsolv 0.7.14 h8b12597_0 conda-forge libssh2 1.9.0 hab1572f_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libtiff 4.1.0 hc3755c2_3 conda-forge libtool 2.4.6 h14c3975_1002 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libwebp 1.0.2 hf4e8a37_4 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxml2 2.9.10 h72b56ed_1 conda-forge linear-tsv 1.1.0 pypi_0 pypi llvm-openmp 10.0.0 hc9558a2_0 conda-forge lz4-c 1.9.2 he1b5a44_1 conda-forge lzo 2.10 h14c3975_1000 conda-forge mafft 7.467 h516909a_0 bioconda mamba 0.3.6 py36h1ccf03e_0 conda-forge markdown 3.2.2 pypi_0 pypi markupsafe 2.0.0a1 pypi_0 pypi matplotlib 3.2.1 pypi_0 pypi minimap2 2.17 h8b12597_1 bioconda mistune 0.8.4 pypi_0 pypi mpi 1.0 mpich conda-forge mpich 3.3.2 hc856adb_0 conda-forge multidict 4.7.5 py36h8c4c3a4_1 conda-forge nbclient 0.4.0 pypi_0 pypi nbconvert 6.0.0a3 pypi_0 pypi nbformat 5.0.6 py_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge nest-asyncio 1.3.3 pypi_0 pypi networkx 2.4 py_1 conda-forge numpy 1.18.5 py36h7314795_0 conda-forge oauth2client 4.1.3 py_0 conda-forge openjpeg 2.3.1 h981e76c_3 conda-forge openpyxl 3.0.3 pypi_0 pypi openssl 1.1.1g h516909a_0 conda-forge packaging 20.4 pypi_0 pypi pandas 1.0.1 pypi_0 pypi pango 1.40.14 he7ab937_1005 conda-forge paramiko 2.7.1 py36_0 conda-forge parso 0.7.0 pypi_0 pypi pcre 8.44 he1b5a44_0 conda-forge perl 5.26.2 h516909a_1006 conda-forge pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pip 19.3.1 py36_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pkg-config 0.29.2 h516909a_1006 conda-forge prettytable 0.7.2 py_3 conda-forge prompt-toolkit 3.0.5 pypi_0 pypi protobuf 3.12.2 pypi_0 pypi psutil 5.7.0 py36h8c4c3a4_1 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 pypi_0 pypi pweave 0.30.3 pypi_0 pypi pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.8 pypi_0 pypi pycosat 0.6.3 py36h8c4c3a4_1004 conda-forge pycountry 19.8.18 pypi_0 pypi pycparser 2.20 py_0 conda-forge pygments 2.6.1 pypi_0 pypi pygraphviz 1.5 py36h8c4c3a4_1002 conda-forge pynacl 1.3.0 py36h516909a_1001 conda-forge pyopenssl 19.1.0 py_1 conda-forge pyparsing 3.0.0a1 pypi_0 pypi pyrsistent 0.16.0 py36h8c4c3a4_0 conda-forge pysam 0.16.0 pypi_0 pypi pysftp 0.2.9 py_1 conda-forge pysocks 1.7.1 py36h9f0ad1d_1 conda-forge python 3.6.10 h8356626_1011_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python-irodsclient 0.8.2 py_0 conda-forge python_abi 3.6 1_cp36m conda-forge pytz 2020.1 pyh9f0ad1d_0 conda-forge pyyaml 5.3.1 py36h8c4c3a4_0 conda-forge pyzmq 19.0.1 pypi_0 pypi ratelimiter 1.2.0 py36h9f0ad1d_1001 conda-forge readline 8.0 hf8c457e_0 conda-forge requests 2.23.0 pyh8c360ce_2 conda-forge rfc3986 1.4.0 pypi_0 pypi rsa 4.6 pyh9f0ad1d_0 conda-forge ruamel_yaml 0.15.80 py36h8c4c3a4_1001 conda-forge s3transfer 0.3.3 pypi_0 pypi scipy 1.4.1 pypi_0 pypi setuptools 47.3.0 py36h9f0ad1d_0 conda-forge simplejson 3.17.0 py36h8c4c3a4_1 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge slacker 0.14.0 py_0 conda-forge smmap 3.0.4 pyh9f0ad1d_0 conda-forge snakemake 5.19.3 0 bioconda snakemake-minimal 5.19.3 py_0 bioconda sqlalchemy 1.3.17 pypi_0 pypi sqlite 3.30.1 hcee41ef_0 conda-forge tableschema 1.19.2 pypi_0 pypi tabulate 0.8.7 pypi_0 pypi tabulator 1.52.1 pypi_0 pypi testpath 0.4.4 pypi_0 pypi tk 8.6.10 hed695b0_0 conda-forge toposort 1.5 py_3 conda-forge tornado 6.0.4 pypi_0 pypi tqdm 4.46.1 pyh9f0ad1d_0 conda-forge traitlets 4.3.3 py36h9f0ad1d_1 conda-forge typing_extensions 3.7.4.2 py_0 conda-forge unicodecsv 0.14.1 pypi_0 pypi uritemplate 3.0.1 py_0 conda-forge urllib3 1.25.9 py_0 conda-forge wcwidth 0.2.4 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.34.2 py_1 conda-forge wrapt 1.12.1 py36h8c4c3a4_1 conda-forge xlrd 1.2.0 pypi_0 pypi xmlrunner 1.7.7 py_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.9 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxext 1.3.4 h516909a_0 conda-forge xorg-libxpm 3.5.13 h516909a_0 conda-forge xorg-libxrender 0.9.10 h516909a_1002 conda-forge xorg-libxt 1.1.5 h516909a_1003 conda-forge xorg-renderproto 0.11.1 h14c3975_1002 conda-forge xorg-xextproto 7.3.0 h14c3975_1002 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.5 h516909a_0 conda-forge yaml 0.2.5 h516909a_0 conda-forge yarl 1.4.2 py36h516909a_0 conda-forge zipp 3.1.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.4 h6597ccf_3 conda-forge `

It seems to me that it should work as it is the write version of snakemake. I'll have a go at re-installing everything.

aineniamh commented 4 years ago

So, it seems you're running a newer version of snakemake than I am, you have:

snakemake 5.19.3 0 bioconda
snakemake-minimal 5.19.3 py_0 bioconda

But I'm running

> snakemake --version
5.13.0

I might suggest installing the exact version specified in the conda environment to see if that solves the issue.

If you try creating the environment with the command here, it should install the correct version of snakemake to the environment.

Let me know if that solves the issue? I'll look into if cores is deprecated in newer releases of snakemake.

josephhughes commented 4 years ago

I re-installed following your installation but snakemake did not install. conda list showed snakemake-minimal 5.13.0 py_0 bioconda but not snakemake.

I got the same "core" error when I re-ran.

So I added snamkemake=5.13 in environment.yml and repeated the installation. Then snakemake did appear with conda list:

snakemake                 5.13.0                        0    bioconda
snakemake-minimal         5.13.0                     py_0    bioconda

However the "core" error still persists.

I don't think it is anything to do with civet, I've always struggled with conda and something weird is going on as when I do snakemake -v, I get 5.5.4 even though I am in the conda environment that lists snakemake 5.13

rambaut commented 4 years ago

Is the 5.5.4 on your PATH earlier than the conda installed one? which snakemake?

aineniamh commented 4 years ago

If you have homebrew installed as well, sometimes there's conflicts.

But if you check your .bash_profile, this should be the final thing in it (or something like this depending on whatever version of conda you installed):

# <<< conda init <<<
# added by Miniconda3 4.7.10 installer
# >>> conda init >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$(CONDA_REPORT_ERRORS=false '/Users/aineniamhotoole/miniconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
    \eval "$__conda_setup"
else
    if [ -f "/Users/aineniamhotoole/miniconda3/etc/profile.d/conda.sh" ]; then
# . "/Users/aineniamhotoole/miniconda3/etc/profile.d/conda.sh"  # commented out by conda initialize
        CONDA_CHANGEPS1=false conda activate base
    else
        \export PATH="/Users/aineniamhotoole/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda init <<<

josephhughes commented 4 years ago

Outwith the civet environment, which snakemake /home1/hugh01j/.local/bin/snakemake /home1/hugh01j/.local/bin/snakemake -v 5.19.3

Within civet, which snakelike /home1/hugh01j/miniconda3/envs/civet/bin/snakemake /home1/hugh01j/miniconda3/envs/civet/bin/snakemake -v 5.5.4

And conda list

snakemake                 5.13.0                        0    bioconda
snakemake-minimal         5.13.0                     py_0    bioconda

And I have the similar conda setup paths at the end of my .bash_profile.

Weird!

aineniamh commented 4 years ago

That is quite mysterious.

When I conda list inside the civet environment, i get

snakemake-minimal         5.13.0                     py_0    bioconda

And no mention of regular snakemake.

I don't have snakemake installed in base, so I might suggest the following steps to help troubleshoot.

Uninstalling the version of snakemake in base environment conda uninstall snakemake
Double check in base that there isn't another snakemake installed somewhere else snakemake -v
Remove the civet environment conda env remove --name civet -y --all or to be sure it's all gone maybe just remove the whole environment directory manually
git pull in civet dir to make sure you've the latest env file
conda env create -f environment.yml
conda activate civet
Check snakemake -v

I would find it very surprising if you still had an odd version of snakemake after that. Alternatively scrap all of miniconda and try again, or manually install the software with the correct versions not using conda. 🤷‍♀️

I hope this helps, conda works in mysterious ways!

josephhughes commented 4 years ago

This is connected to my environment set-up, I haven't figured out what is wrong yet but nothing to do with civet.

artic-network / civet

Test data #17

Name Version Build Channel