No module named 'pandas._libs.tslibs.conversion'

[slambrechts]

Hi Alex,

I seem to get a few ModuleNotFoundError: No module named 'pandas._libs.tslibs.conversion' errors while running snakemake --use-conda -k -j 48

And it seems to be while the pipeline calls VIBRANT (see below)

I'm using the test input samples, and virsorter2 and deepvirfinder give viral sequences as output, but not vibrant.

Did you experience this?

Traceback (most recent call last):
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/__init__.py", line 30, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/_libs/__init__.py", line 3, in <module>
    from .tslibs import (
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/_libs/tslibs/__init__.py", line 3, in <module>
    from .conversion import localize_pydatetime, normalize_date
ModuleNotFoundError: No module named 'pandas._libs.tslibs.conversion'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/bin/VIBRANT_run.py", line 22, in <module>
    import pandas as pd
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/__init__.py", line 38, in <module>
    "the C extensions first.".format(module)
ImportError: C extension: No module named 'pandas._libs.tslibs.conversion' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.
Traceback (most recent call last):
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/__init__.py", line 30, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/_libs/__init__.py", line 3, in <module>
    from .tslibs import (
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/_libs/tslibs/__init__.py", line 3, in <module>
    from .conversion import localize_pydatetime, normalize_date
ModuleNotFoundError: No module named 'pandas._libs.tslibs.conversion'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/bin/VIBRANT_run.py", line 22, in <module>
    import pandas as pd
  File "/kyukon/scratch/gent/vo/000/gvo00043/vsc42339/virsearch/.snakemake/conda/76fccd69f65a29a0e12811501bdb0bc7/site-packages/pandas/__init__.py", line 38, in <module>
    "the C extensions first.".format(module)
ImportError: C extension: No module named 'pandas._libs.tslibs.conversion' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.

[alexmsalmeida]

Hi Sam,

I haven't come across this issue before, but a quick google search suggests that it may be an issue with your local pandas installation. As suggested here you could try uninstalling and reinstalling pandas. Otherwise, you might try running the snakemake workflow in a new clean conda env.

Alex

[slambrechts]

Hi Alex,

For virsearch I use snakemake in a separate conda env, and installed it like so:

mamba create -c conda-forge -c bioconda -n snakemake snakemake=6.3

So the conda env is new and should be clean. Or do you use a central snakemake installation?

Uninstalling and reinstalling pandas did not solve the problem unfortunately.

What version of pandas do you use?

Best, Sam

[alexmsalmeida]

Hi Sam,

After a bit of troubleshooting it seems the issue is related with the numpy version the conda environment is using. I downgraded numpy from v1.19 to 1.17 and it seems to have solved the issue. Not really sure why this error only popped up now, as it seemed to be working before.

Anyway, I've updated the envs/vibrant.yml file, so either reclone or update the file and it should work now.

Best, Alex

[slambrechts]

Hi Alex,

Great! I was trying different versions of pandas, but hadn't tried numpy yet. It indeed solves this issue, thanks!

Is this the correct output for the test samples you provide with the repo:

Sample 1:

Sequence_ID Order   Percent_of_votes    Family  Percent_of_votes
DVF_NODE_210_length_49919_cov_4.803225_1    Caudovirales    100 Siphoviridae    60
DVF_NODE_45_length_101474_cov_12.816711 Caudovirales    50  Siphoviridae    60.9756097560976
VIBRANT_NODE_39_length_106331_cov_11.393786_fragment_2_1    Caudovirales    100 Siphoviridae    75
VIBRANT_NODE_450_length_33903_cov_33.374261_1   Caudovirales    100 Siphoviridae    76.9230769230769
VS2_NODE_134_length_62313_cov_15.345594||full_1 Caudovirales    100 Podoviridae 44

Sample 2 and 3:

no high-confidence viral sequences?

[alexmsalmeida]

Hi Sam,

Excellent, glad it's sorted! Yup, can confirm those are the expected results.

Alex