Failed to import module (No module named '_cgatools')

[smcgrat]

Hello,

Firstly, I've tried searching for a solution to this but didn't find anything. Also, if I have missed something from the documents that would avoid this please accept my apologies.

The error I am encountering is as follows:

$ /home/shared/apps/apps/variant_tools/python-3.7.0/dev/20191015/bin/vtools show pipelines
Failed to import module (No module named '_cgatools')
Please verify if you have installed variant tools successfully (using command "python setup.py install")

The installation steps followed where:

$ git clone https://github.com/vatlab/varianttools.git varianttools
$ cd varianttools
$ export PYTHONPATH=/home/shared/apps/apps/variant_tools/python-3.7.0/dev/20191015/python_lib
$ python setup.py install --install-platlib=/home/shared/apps/apps/variant_tools/python-3.7.0/dev/20191015/python_lib --install-scripts=/home/shared/apps/apps/variant_tools/python-3.7.0/dev/20191015/bin

Python version in use

$ python --version
Python 3.7.0

Any pointers on what I am doing wrong would be much appreciated.

Thanks

Sean

[BoPeng]

Did you try our conda release?

[smcgrat]

Thanks BoPeng,

Sorry this took so long to get back on.

I have installed vtools using conda but am getting the same error as follows:

(vtools) [nighralf@jal ~]$ which vtools
~/.conda/envs/vtools/bin/vtools
(vtools) [nighralf@jal ~]$ vtools show pipelines
Failed to import module (No module named '_cgatools')
Please verify if you have installed variant tools successfully (using command "python setup.py install")

I'd appreciate any assistance that can be offered please.

Thanks

Sean

[BoPeng]

Are you on a linux system? Is it a Ubuntu or RHEL/CentOS system?

[smcgrat]

We're using Scientific Linux release 7.7. Its a derivative of CentOS. Thanks Sean

[BoPeng]

Thanks for the info. I will try to reproduce and fix it today, and make a new conda-forge release

[BoPeng]

This has been fixed in the latest master along with (#122) but we are not sure if variant tools works as expected.

Installation instruction (temporary)

1. install gsl and boost libraries and needed library to compile variant tools

   conda install numpy Cython gsl libboost

2. Set LD_INCLUDE_PATH to conda include, something like the following (replace vtools with your own environment)

   export LD_INCLUDE_PATH=/home/bpeng1/anaconda3/envs/vtools/include

3. Compile with

python setup.py install

or

pip install -e .

[smcgrat]

Hi Bo,

Thanks for that. I'm getting an error when I run vtools now unfortunately.

This is the install process used:

$ module load python-3.7.0-gcc-8.2.0-g4ikncu
$ conda create --name vtools3 python=3
$ conda activate vtools3
$ conda install numpy Cython gsl libboost
$ export LD_INCLUDE_PATH=/home/users/username/.conda/envs/vtools3/include
$ cd /home/users/username/.conda/envs/vtools3/
$ git clone https://github.com/vatlab/varianttools.git varianttools
$ cd varianttools/
$ python setup.py install

This is the error I get when I try to run vtools:

(vtools3) [nighralf@jal varianttools]$ vtools show pipelines
Traceback (most recent call last):
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 18, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/users/nighralf/.conda/envs/vtools3/lib/libgsl.so.23: undefined symbol: cblas_dasum

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/users/nighralf/.conda/envs/vtools3/bin/vtools", line 11, in <module>
    load_entry_point('variant-tools==3.0.6', 'console_scripts', 'vtools')()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/vtools.py", line 53, in <module>
    import variant_tools.association as association
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/association.py", line 35, in <module>
    from .assoTests import AssoData
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 21, in <module>
    _assoTests = swig_import_helper()
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 20, in swig_import_helper
    return importlib.import_module('_assoTests')
  File "/home/users/nighralf/.conda/envs/vtools3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_assoTests'

Sorry if I have missed something or if this is a peculiarity of our setup. I'd be very grateful for any advice you can offer please.

Thanks

Sean

[BoPeng]

It seems that your system libgsl needs libcblas, could you please do

conda install libcblas

and, in this line of code, add blas or cblas back, and let us know if it works?

[smcgrat]

Thanks for that Bo and your continuing help, its very good of you and I'm very grateful. Unfortunately I'm still getting the same error though.

Here's what I did.

1. Using blas

   $ conda config --add channels bioconda
   $ conda config --add channels conda-forge
   $ conda install libcblas
   $ diff -u orig-setup.py setup.py
   ... (SNIP) ...
   -            libraries = libs + ['gsl', 'stat'], #, 'blas'],
   +            libraries = libs + ['gsl', 'stat', 'blas'],
   $ python setup.py clean
   $ python setup.py install

2. Using cblas Same as above but with the following change in the setup.py

   -            libraries = libs + ['gsl', 'stat'], #, 'blas'],
   +            libraries = libs + ['gsl', 'stat', 'cblas'],

Running vtools then gives this error for both of those changes:

$ vtools
Traceback (most recent call last):
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 18, in swig_import_helper
    return importlib.import_module(mname)
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/users/nighralf/.conda/envs/vtools4/lib/libgsl.so.23: undefined symbol: cblas_dasum

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/users/nighralf/.conda/envs/vtools4/bin/vtools", line 11, in <module>
    load_entry_point('variant-tools==3.0.6', 'console_scripts', 'vtools')()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/home/users/nighralf/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/vtools.py", line 53, in <module>
    import variant_tools.association as association
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/association.py", line 35, in <module>
    from .assoTests import AssoData
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 21, in <module>
    _assoTests = swig_import_helper()
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant_tools-3.0.6-py3.7-linux-x86_64.egg/variant_tools/assoTests.py", line 20, in swig_import_helper
    return importlib.import_module('_assoTests')
  File "/home/users/nighralf/.conda/envs/vtools4/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_assoTests'

Is there something I am missing or doing wrong here? Sorry if so.

Thanks

Sean

[BoPeng]

Did you remove dist and build, and run python setup.py install again after the change? I would even remove /home/users/nighralf/.conda/envs/vtools4/lib/python3.7/site-packages/variant* to make sure everything is freshly built.

If this is not the problem, please send me the exact versions of boost, gsl, blas etc, greped from the output of conda list. I built vtools from a Scientific Linux 7.7 VM yesterday and everything seemed to be ok. I was using a fresh, therefore latest version of anaconda and everything though.

[smcgrat]

Hi Bo. Thanks for that. You where right, I wasn't clearing up the build and re-creating it properly. I did the following:

$ rm -rf dist/*; rm -rf build/*; rm -rf ~/.conda/envs/vtools/lib/python3.7/site-packages/variant*; python setup.py clean

And the version of vtools that python setup.py install is now creating no longer has those errors once the blas library is included.

Many thanks for all your help and sorry for the hassle. You where very good.

Best

Sean

[BoPeng]

What change did you make to setup.py? Did you add blas or cblas?

[smcgrat]

Sorry, should have mentioned that. Changed it to blas

-            libraries = libs + ['gsl', 'stat'], #, 'blas'],
+            libraries = libs + ['gsl', 'stat', 'blas'],

[BoPeng]

Thanks. I have added blas back.