mob_cluster own db matrix problem

[Lneffe]

Hi! I am tyring to construct a database out of my own plasmids using the command mob_cluster --mode build --infile --outdir Even when trying to run this on the original ncbi_plasmid_full_seqs.fas file I encountered an error.

2020-05-13 13:24:45,779 INFO: Running Mob-Suite Clustering toolkit v. 2.0.0 [in /home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/mob_cluster.py:225] 2020-05-13 13:24:45,779 INFO: Processing fasta file /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/db_test/test_old_db_original/ncbi_plasmid_full_seqs.fas [in /home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/mob_cluster.py:226] 2020-05-13 13:24:45,779 INFO: Analysis directory /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/db_test/test_old_db_original/ [in /home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/mob_cluster.py:227] 2020-05-13 13:55:28,055 INFO: b'' [in /home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/wrappers/init.py:51] Traceback (most recent call last): File "/home/lneffe/miniconda3/bin/mob_cluster", line 10, in sys.exit(main()) File "/home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/mob_cluster.py", line 308, in main clust_assignments = build_cluster_db(distance_matrix_file, (0.05, 0.0001)) File "/home/lneffe/miniconda3/lib/python3.7/site-packages/mob_suite/mob_cluster.py", line 81, in build_cluster_db distance_matrix = data.as_matrix() File "/home/lneffe/miniconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5274, in getattr return object.getattribute(self, name) AttributeError: 'DataFrame' object has no attribute 'as_matrix'

I installed mob-suite with bioconda in a python3.7 environment. Maybe there is something wrong with my installation?

Mayn thanks in advance!

[jrober84]

There is a later version of MOB-suite v. 2.1.0 which I recommend updating to as there was a bug in 2.0.0 which presents some issues with the host-range prediction. I know ".as_matrix()" is now deprecated in pandas and v. 3.0.0 removes this function call. In the short term, I would recommend that you create a fresh conda environment and install MOB-suite through conda so that the versions of each package should match.

[Lneffe]

thank you! I followed your recommendation and set up a new conda environment with python3.7. Next I tried to install the newest mob_suite but I have trouble because it is apparently not the default in the conda channel. Then I tried to run: conda install mob_suite=2.1.0 Which gave me the following error:

UnsatisfiableError: The following specifications were found to be incompatible with each other: Output in format: Requested package -> Available versions

Also I wasn´t able to install it from the .tar.bz2 archive, which I downloaded and gave it a try manually.

conda install "/home/lneffe/miniconda3/envs/mob/bin/mob_suite-2.1.0-py_0.tar.bz2"

Downloading and Extracting Packages ###################################################################################################################################################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done ERROR conda.core.link:_execute(700): An error occurred while installing package '::mob_suite-2.1.0-py_0'. Rolling back transaction: done

LinkError: post-link script failed for package ::mob_suite-2.1.0-py_0 location of failed script: /home/lneffe/miniconda3/envs/mob/bin/.mob_suite-post-link.sh ==> script messages <==

==> script output <== stdout: stderr: Traceback (most recent call last): File "/home/lneffe/miniconda3/envs/mob/bin/mob_init", line 7, in from mob_suite.mob_init import main File "/home/lneffe/miniconda3/envs/mob/lib/python3.7/site-packages/mob_suite/mob_init.py", line 3, in import os, pycurl, tarfile, zipfile, gzip, multiprocessing, sys ModuleNotFoundError: No module named 'pycurl' How can I install this version via conda? best wishes, Lisa

[kbessonov1984]

@Lneffe Hello! Conda can throw some weird errors. The error indicates that pycurl is not installed on the system. I had issues installing pycurl myself because it needs curl-dev packages and gcc compiler on the system. Try the following commands to solve install issue. Also try not to cancel conda install as it can partially damage conda install.

#make conda version resolution more flexible
conda config --set allow_conda_downgrades true
#add all channels
conda config --add channels conda-forge
conda config --add channels bioconda

#try  to install pycurl separately although it should of been installed automatically
conda install pycurl

#install mob-suite
conda install -c bioconda mob_suite==2.1.0 -y

Hopefully it will work. If not, let us know

[Lneffe]

Hi @kbessonov1984 ! Thanks, I followed your advice. I successfully download the pycurl package and installed it. Here is the shell output I observed:

(mob) lneffe@bioinf001:~$ unset PYTHONPATH
(mob) lneffe@bioinf001:~$ conda config --set allow_conda_downgrades true
(mob) lneffe@bioinf001:~$ conda config --add channels conda-forge
Warning: 'conda-forge' already in 'channels' list, moving to the top
(mob) lneffe@bioinf001:~$ conda config --add channels bioconda
Warning: 'bioconda' already in 'channels' list, moving to the top
---
# installed pycurl 
---
(mob) lneffe@bioinf001:~$ conda install -c bioconda mob_suite==2.1.0 -y
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.                                                                                                                                               failed                                                                                                                                                                                              -
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions

There still seems to be an issue with obtaining the up to date package via conda.

Thanks a lot for your help, Lisa

[kbessonov1984]

Hi Lisa, It looks like an issue with conda installation itself. There are reports that indicate some incompatibilities between package versions, python version and conda. My adivice is to create a new environment and downgrade to conda 4.6, although in VM, I was able to install mob-suite 2.0.1 running conda version 4.8.2

conda install conda=4.6.14 -y
conda create -n mob-suite-2.0.1 python=3.6 -y
conda activate mob-suite-2.0.1 
conda install -c bioconda mob_suite==2.1.0 -y

You can also try to run conda clean --all -y to remove old packages and free up some space. Alternatively we have a docker image that you can use if conda would not work for you.

Also you can try to install mob-suite via pip inside conda environment with pycurl installed

conda activate mob-suite-2.0.1
conda install pycurl mash blast #install key dependencies
pip install mob_suite==2.1.0

[Lneffe]

Hi Kirill! Thank you, I finally managed to get the latest mob-suite version installed. Ended up using pip installation and adding the required packages manually with conda. But now unfortunately the same error occurs, when I try to generate a database from a multifasta file

(mob-suite) lneffe@bioinf001:~$ mob_cluster --mode build --infile /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/native_eco_bs_00010_assembly_plasmids.fasta --outdir /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/db_test/db_native_eco_bs_10/ 2020-05-14 16:17:01,318 INFO: Running Mob-Suite Clustering toolkit v. 2.1.0 [in /home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py:226] 2020-05-14 16:17:01,318 INFO: Processing fasta file /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/native_eco_bs_00010_assembly_plasmids.fasta [in /home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py:227] 2020-05-14 16:17:01,318 INFO: Analysis directory /vol/projects/MOBA/eco_bs_clin_iso/plasmid_typing_annotation/db_test/db_native_eco_bs_10/ [in /home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py:228] 2020-05-14 16:17:01,819 INFO: b'' [in /home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/wrappers/__init__.py:24] Traceback (most recent call last): File "/home/lneffe/miniconda3/envs/mob-suite/bin/mob_cluster", line 8, in <module> sys.exit(main()) File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py", line 309, in main clust_assignments = build_cluster_db(distance_matrix_file, (0.05, 0.0001)) File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py", line 82, in build_cluster_db distance_matrix = data.as_matrix() File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/generic.py", line 5274, in __getattr__ return object.__getattribute__(self, name) AttributeError: 'DataFrame' object has no attribute 'as_matrix' Best Wishes, Lisa

[kbessonov1984]

Hi Lise, The issue is with the pandas. as_matrix method is deprecated since 0.23.0 pandas version (see here). Try to install pandas version 0.23.0 and it should resolve your issue like so

conda install -c conda-forge pandas==0.23.0 -y
#if conda environment resolution will fail, try via pip
pip install pandas==0.23.0

[Lneffe]

Hi, thank you. the installation of pandas 0.23.0 into the environment worked well. When trying to run it now I encoutered a new error:

Traceback (most recent call last): File "/home/lneffe/miniconda3/envs/mob-suite/bin/mob_cluster", line 5, in <module> from mob_suite.mob_cluster import main File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/mob_suite/mob_cluster.py", line 7, in <module> import pandas as pd File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module> from pandas.core.api import * File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module> from pandas.core.groupby.groupby import Grouper File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 2, in <module> from pandas.core.groupby.groupby import ( File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 46, in <module> from pandas.core.index import (Index, MultiIndex, File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/index.py", line 2, in <module> from pandas.core.indexes.api import * File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/indexes/api.py", line 4, in <module> from pandas.core.indexes.base import (Index, File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 52, in <module> from pandas.core import ops File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/ops/__init__.py", line 25, in <module> from pandas.core.construction import extract_array File "/home/lneffe/miniconda3/envs/mob-suite/lib/python3.6/site-packages/pandas/core/construction.py", line 13, in <module> from pandas._libs.tslibs import IncompatibleFrequency, OutOfBoundsDatetime ImportError: cannot import name 'IncompatibleFrequency' This seems still be an issue with the pandas package. Do you have an idea how to fix it?

Thanks!

[kbessonov1984]

Looks like something weird have happened with pandas package so it can not import some of its sub-modules. I would try to re-install pandas package 0.23.0 either via conda or via pip pip install --upgrade --force-reinstall pandas==0.23.0. Alternatively, I would create a new fresh conda environment and would install pandas and pycurl packages via conda and all the rest via pip. Once you install pandas try to run python -c "import pandas" if you see no message then it means that it is installed correctly. Also try to run mob_cluster -h and you should see command help information.

[Lneffe]

Thank you for your good support! The whole tool works now!

Best wishes, Lisa