search

Psy-Fer / deeplexicon

Signal based nanopore RNA demultiplexing with convolutional neural networks
https://psy-fer.github.io/deeplexicon/
MIT License
34 stars 8 forks source link

errors with deeplexicon and empty output #21

Open ogrecio opened 2 years ago

ogrecio commented 2 years ago

Hi! I am ghaving an error that I do not understand from deeplexicon.py. The output is empty. Could you please let me know what's going on and how to solve this issue? Thank you!

[...]
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_896e4350-9eef-4ecc-a41e-a81f72d97076
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_8999db3a-34ff-43b1-b147-3d7a39d97ab2
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_89b92d60-4594-4bb0-84f0-b2f640dc0fa1
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_89bb0964-9c58-4259-8164-b715dde99ebd
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_89cbf87d-71e2-4f16-99f5-d65c31c52697
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_89db305c-e504-4a35-b113-9cb0dd127027
Traceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)
error: extract_fast5():failed to read readID: read_8a017ab4-16c8-460d-8f8b-c6fb3c1ebd58
Traceback (most recent call last):
^CTraceback (most recent call last):
  File "deeplexicon.py", line 430, in read_multi_fast5
    for col in hdf[read]['Raw/Signal'][()]:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 573, in __getitem__
    self.id.read(mspace, fspace, arr, mtype, dxpl=self._dxpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read
  File "h5py/_proxy.pyx", line 130, in h5py._proxy.dset_rw
  File "h5py/_proxy.pyx", line 84, in h5py._proxy.H5PY_H5Dread
OSError: Can't read data (can't open directory: /usr/local/hdf5/lib/plugin)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deeplexicon.py", line 622, in <module>
    main()
  File "deeplexicon.py", line 278, in main
    seg_signal = get_multi_fast5_signal(fast5_file, window, squig_file, seg_file)
  File "deeplexicon.py", line 357, in get_multi_fast5_signal
    f5_dic = read_multi_fast5(read_filename)
  File "deeplexicon.py", line 439, in read_multi_fast5
    traceback.print_exc()
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/usr/lib/python3.7/traceback.py", line 163, in print_exc
    print_exception(*sys.exc_info(), limit=limit, file=file, chain=chain)
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/usr/lib/python3.7/traceback.py", line 104, in print_exception
    type(value), value, tb, limit=limit).format(chain=chain):
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/usr/lib/python3.7/traceback.py", line 610, in format
    yield from self.stack.format()
  File "/mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/usr/lib/python3.7/traceback.py", line 421, in format
    frame.filename, frame.lineno, frame.name))

[...]

info: reading sig_count: 3994/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 3995/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 3996/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 3997/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 3998/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 3999/4000
info: dRNA_segmenter: no seg found:
info: reading sig_count: 4000/4000
info: dRNA_segmenter: no seg found:
Traceback (most recent call last):
  File "deeplexicon.py", line 622, in <module>
    main()
  File "deeplexicon.py", line 304, in main
    C = classify(model, labels, np.array(images), False, args.threshold)
  File "deeplexicon.py", line 611, in classify
    y = model.predict(x, verbose=0)
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/keras/engine/training.py", line 1149, in predict
    x, _, _ = self._standardize_user_data(x)
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/keras/engine/training.py", line 751, in _standardize_user_data
    exception_prefix='input')
  File "/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/keras/engine/training_utils.py", line 128, in standardize_input_data
    'with shape ' + str(data_shape))
ValueError: Error when checking input: expected input_1 to have 4 dimensions, but got array with shape (0, 1)
Psy-Fer commented 2 years ago

Hello,

That looks like your fast5 files have VBZ compression on them.

Please follow the instructions noted here on squigglekit to install the vbz plugin for the H5py library to decompress the data.

https://github.com/Psy-Fer/SquiggleKit#vbz-compressed-fast5

Let me know how you get along.

James

ogrecio commented 2 years ago

Hi James, thanks for your quick response. Apparently I have issues with my python version and the packages versions. I'm working on an HPC and I need to load either of these versions: python/2.7.18 python/3.6.12 python/3.9.9

All of them are giving me incompatibilities with the necessary packages. An the execution ends up in error messages like this. Traceback (most recent call last): File "/mnt/lustre/scratch/nlsas/home/csic/mgr/ogr/programs/deeplexicon/deeplexicon.py", line 20, in <module> from pyts.image import MarkovTransitionField, GramianAngularField, RecurrencePlot File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/pyts/image/__init__.py", line 6, in <module> from .gaf import GramianAngularField File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/pyts/image/gaf.py", line 8, in <module> from numba import njit, prange File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/__init__.py", line 45, in <module> import numba.typed File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/typed/__init__.py", line 3, in <module> from .typeddict import Dict File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/typed/typeddict.py", line 19, in <module> def _make_dict(keyty, valty): File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/decorators.py", line 224, in njit return jit(*args, **kws) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/decorators.py", line 161, in jit return wrapper(pyfunc) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/decorators.py", line 175, in wrapper disp = dispatcher(py_func=func, locals=locals, File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/dispatcher.py", line 576, in __init__ self.targetctx = self.targetdescr.target_context File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/registry.py", line 50, in target_context return self._toplevel_target_context File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/utils.py", line 381, in __get__ res = instance.__dict__[self.name] = self.func(instance) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/registry.py", line 34, in _toplevel_target_context return cpu.CPUContext(self.typing_context) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/base.py", line 250, in __init__ self.init() File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/compiler_lock.py", line 32, in _acquire_compile_lock return func(*args, **kwargs) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/cpu.py", line 49, in init self._internal_codegen = codegen.JITCPUCodegen("numba.exec") File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/codegen.py", line 612, in __init__ self._init(self._llvm_module) File "/home/csic/mgr/ogr/.local/lib/python3.9/site-packages/numba/targets/codegen.py", line 621, in _init tm = target.create_target_machine(**tm_options) TypeError: create_target_machine() got an unexpected keyword argument 'jitdebug'

Could you provide any tip before I contact the IT from administrating the supercomputer?

Thanks!

Psy-Fer commented 2 years ago

Hey,

i'd try with python/3.9.9 first, as I recommend 3.7, the other 2 won't work so well.

The biggest issue is the versions of the other packages. Things have gotten better with the new tensorflow, but we have not done the vigorous work to convert and then validate upgrading to tensorlfow 2.0+ to make this easier. (sorry, just been busy with other projects, I'm sure you know how it is)

For python3.9, could you give me the errors you are getting for the packages? I can try and guide you through those (and reproducing them on my side should be easier if I know what to be looking for)

ogrecio commented 2 years ago

I have dealt with some the packages, but for instance this is creting some trouble:

python3 deeplexicon.py -p /fast5_pass/ -f multi -m  models/pAmps-final-actrun_newdata_nanopore_UResNet20v2_model.030.h5 > output.tsv
Traceback (most recent call last):
  File "deeplexicon.py", line 25, in <module>
    from keras.optimizers import Adam
ImportError: cannot import name 'Adam' from 'keras.optimizers' (/home/csic/mgr/ogr/.local/lib/python3.7/site-packages/keras/optimizers.py)

Also the instruction to install install the vbz plugin for the H5py library to decompress the data are not clear in their github. The h5py modelu in my HPC is installed for python 3.6, so I need to install it at the user level. Do you know where are step by step instructions to install the plugin for vbz decompressing?

Psy-Fer commented 2 years ago

Could you please do a

pip freeze

So I can see the versions of everything?

For vbz plugin, it's managed by ONT. They have a number of ways to install in their releases page. Ideally you would want to be using a virtual environment for python so you can control the package installations. Then use either a .whl file for the appropriate python version, or get the source and do an install locally into the virtual env. To do that, have your env active, then untar the the source dist then do a python setup.py install and it should automatically install into your H5py lib.

Running things in a virtual env is a much easier way to get things working then trying to deal with the main install of python on the system.

To make a new venv, just do

python3 -m venv newvenv
source newvenv/bin/activate
python3 -m pip --upgrade install

Then yoh should be ably to install all the requirements to that venv with pip, then do the vbz install.

(let me know if you have any troubles with that. On my phone at the moment, but wanted to get back you while you seem to be working on it)

Cheers

ogrecio commented 2 years ago

James, thank you so much for helping me out with this. I will work with an environment. I am not able to install the pluging from the source code. Apparently no setup.py file is provided, (sorry I'm a python dummy). But I can try to load an existing module load h5py/3.6.0-python-3.9.9, and see if it works

This is the result of pip freeze

alabaster @ file:///dev/shm/vis.software/Python/3.9.9/system-system/alabaster/alabaster-0.7.12
appdirs @ file:///dev/shm/vis.software/Python/3.9.9/system-system/appdirs/appdirs-1.4.4
argcomplete @ file:///dev/shm/vis.software/Python/3.9.9/system-system/argcomplete/argcomplete-1.12.3
argon2-cffi @ file:///dev/shm/vis.software/Python/3.9.9/system-system/argon2cffi/argon2-cffi-21.1.0
asn1crypto @ file:///dev/shm/vis.software/Python/3.9.9/system-system/asn1crypto/asn1crypto-1.4.0
async-generator @ file:///dev/shm/vis.software/Python/3.9.9/system-system/async_generator/async_generator-1.10
atomicwrites @ file:///dev/shm/vis.software/Python/3.9.9/system-system/atomicwrites/atomicwrites-1.4.0
attrs @ file:///dev/shm/vis.software/Python/3.9.9/system-system/attrs/attrs-21.2.0
Babel @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Babel/Babel-2.9.1
backcall @ file:///dev/shm/vis.software/Python/3.9.9/system-system/backcall/backcall-0.2.0
backports.entry-points-selectable @ file:///dev/shm/vis.software/Python/3.9.9/system-system/backportsentry_points_selectable/backports.entry_points_selectable-1.1.1
backports.functools-lru-cache @ file:///dev/shm/vis.software/Python/3.9.9/system-system/backportsfunctools_lru_cache/backports.functools_lru_cache-1.6.4
bcrypt @ file:///dev/shm/vis.software/Python/3.9.9/system-system/bcrypt/bcrypt-3.2.0
beniget @ file:///dev/shm/vis.software/Python/3.9.9/system-system/beniget/beniget-0.4.1
bitstring @ file:///dev/shm/vis.software/Python/3.9.9/system-system/bitstring/bitstring-3.1.9
bleach @ file:///dev/shm/vis.software/Python/3.9.9/system-system/bleach/bleach-4.1.0
blist @ file:///dev/shm/vis.software/Python/3.9.9/system-system/blist/blist-1.3.6
bottle==0.12.19
CacheControl @ file:///dev/shm/vis.software/Python/3.9.9/system-system/CacheControl/CacheControl-0.12.6
cached-property @ file:///dev/shm/vis.software/Python/3.9.9/system-system/cachedproperty/cached-property-1.5.2
cachy @ file:///opt/cesga/2020/sources/p/Python/extensions/cachy-0.3.0-py2.py3-none-any.whl
certifi @ file:///dev/shm/vis.software/Python/3.9.9/system-system/certifi/certifi-2021.10.8
cffi @ file:///dev/shm/vis.software/Python/3.9.9/system-system/cffi/cffi-1.15.0
chardet @ file:///dev/shm/vis.software/Python/3.9.9/system-system/chardet/chardet-4.0.0
charset-normalizer @ file:///dev/shm/vis.software/Python/3.9.9/system-system/charsetnormalizer/charset-normalizer-2.0.7
cleo @ file:///opt/cesga/2020/sources/p/Python/extensions/cleo-0.8.1-py2.py3-none-any.whl
click @ file:///dev/shm/vis.software/Python/3.9.9/system-system/click/click-8.0.3
clikit @ file:///opt/cesga/2020/sources/p/Python/extensions/clikit-0.6.2-py2.py3-none-any.whl
cmake==3.22.0
colorama @ file:///dev/shm/vis.software/Python/3.9.9/system-system/colorama/colorama-0.4.4
conan==1.43.0
contextvars @ file:///dev/shm/vis.software/Python/3.9.9/system-system/contextvars/contextvars-2.4
crashtest @ file:///opt/cesga/2020/sources/p/Python/extensions/crashtest-0.3.1-py3-none-any.whl
cryptography @ file:///dev/shm/vis.software/Python/3.9.9/system-system/cryptography/cryptography-36.0.0
Cython @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Cython/Cython-0.29.24
DateTime @ file:///dev/shm/vis.software/Python/3.9.9/system-system/DateTime/DateTime-4.3
debugpy @ file:///opt/cesga/2020/sources/p/Python/extensions/debugpy-1.5.1-py2.py3-none-any.whl
decorator @ file:///dev/shm/vis.software/Python/3.9.9/system-system/decorator/decorator-5.1.0
defusedxml @ file:///dev/shm/vis.software/Python/3.9.9/system-system/defusedxml/defusedxml-0.7.1
deprecation @ file:///dev/shm/vis.software/Python/3.9.9/system-system/deprecation/deprecation-2.1.0
dill @ file:///opt/cesga/2020/sources/p/Python/extensions/dill-0.3.4-py2.py3-none-any.whl
distlib @ file:///dev/shm/vis.software/Python/3.9.9/system-system/distlib/distlib-0.3.3
distro @ file:///dev/shm/vis.software/Python/3.9.9/system-system/distro/distro-1.6.0
docopt @ file:///dev/shm/vis.software/Python/3.9.9/system-system/docopt/docopt-0.6.2
docutils @ file:///dev/shm/vis.software/Python/3.9.9/system-system/docutils/docutils-0.17.1
ecdsa @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ecdsa/ecdsa-0.17.0
entrypoints @ file:///dev/shm/vis.software/Python/3.9.9/system-system/entrypoints/entrypoints-0.3
et-xmlfile @ file:///dev/shm/vis.software/Python/3.9.9/system-system/et_xmlfile/et_xmlfile-1.1.0
fasteners==0.16.3
fastjsonschema @ file:///dev/shm/vis.software/Python/3.9.9/system-system/fastjsonschema/fastjsonschema-2.15.1
filelock @ file:///dev/shm/vis.software/Python/3.9.9/system-system/filelock/filelock-3.4.0
flit @ file:///dev/shm/vis.software/Python/3.9.9/system-system/flit/flit-3.5.0
flit_core @ file:///dev/shm/vis.software/Python/3.9.9/system-system/flitcore/flit_core-3.5.0
fonttools @ file:///dev/shm/vis.software/Python/3.9.9/system-system/fonttools/fonttools-4.28.2
fsspec @ file:///dev/shm/vis.software/Python/3.9.9/system-system/fsspec/fsspec-2021.11.0
funcsigs @ file:///dev/shm/vis.software/Python/3.9.9/system-system/funcsigs/funcsigs-1.0.2
future @ file:///dev/shm/vis.software/Python/3.9.9/system-system/future/future-0.18.2
gast @ file:///dev/shm/vis.software/Python/3.9.9/system-system/gast/gast-0.5.3
gitdb @ file:///dev/shm/vis.software/Python/3.9.9/system-system/gitdb/gitdb-4.0.9
GitPython @ file:///dev/shm/vis.software/Python/3.9.9/system-system/GitPython/GitPython-3.1.24
glob2 @ file:///dev/shm/vis.software/Python/3.9.9/system-system/glob2/glob2-0.7
glpk @ file:///dev/shm/vis.software/Python/3.9.9/system-system/glpk/glpk-0.4.6
h5py @ file:///dev/shm/vis.software/h5py/3.6.0/system-system-python-3.9.9/h5py-3.6.0
html5lib @ file:///dev/shm/vis.software/Python/3.9.9/system-system/html5lib/html5lib-1.1
idna @ file:///dev/shm/vis.software/Python/3.9.9/system-system/idna/idna-3.3
imagesize @ file:///dev/shm/vis.software/Python/3.9.9/system-system/imagesize/imagesize-1.3.0
immutables==0.16
importlib-metadata @ file:///opt/cesga/2020/sources/p/Python/extensions/importlib_metadata-4.8.2-py3-none-any.whl
importlib-resources @ file:///opt/cesga/2020/sources/p/Python/extensions/importlib_resources-5.4.0-py3-none-any.whl
inflection @ file:///dev/shm/vis.software/Python/3.9.9/system-system/inflection/inflection-0.5.1
iniconfig @ file:///dev/shm/vis.software/Python/3.9.9/system-system/iniconfig/iniconfig-1.1.1
intervaltree @ file:///dev/shm/vis.software/Python/3.9.9/system-system/intervaltree/intervaltree-3.1.0
intreehooks @ file:///dev/shm/vis.software/Python/3.9.9/system-system/intreehooks/intreehooks-1.0
ipaddress @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ipaddress/ipaddress-1.0.23
ipykernel @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ipykernel/ipykernel-6.5.1
ipython @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ipython/ipython-7.29.0
ipython-genutils @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ipython_genutils/ipython_genutils-0.2.0
ipywidgets @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ipywidgets/ipywidgets-7.6.5
itsdangerous @ file:///dev/shm/vis.software/Python/3.9.9/system-system/itsdangerous/itsdangerous-2.0.1
jdcal @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jdcal/jdcal-1.4.1
jedi @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jedi/jedi-0.18.1
jeepney @ file:///opt/cesga/2020/sources/p/Python/extensions/jeepney-0.7.1-py3-none-any.whl
Jinja2==2.11.3
joblib @ file:///dev/shm/vis.software/Python/3.9.9/system-system/joblib/joblib-1.1.0
jsonschema @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jsonschema/jsonschema-3.2.0
jupyter-client @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jupyter_client/jupyter_client-7.0.6
jupyter-core @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jupytercore/jupyter_core-4.9.1
jupyter-packaging @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jupyter_packaging/jupyter_packaging-0.11.1
jupyterlab-pygments @ file:///dev/shm/vis.software/Python/3.9.9/system-system/jupyterlab_pygments/jupyterlab_pygments-0.1.2
jupyterlab-widgets @ file:///opt/cesga/2020/sources/p/Python/extensions/jupyterlab_widgets-1.0.2-py3-none-any.whl
keyring @ file:///dev/shm/vis.software/Python/3.9.9/system-system/keyring/keyring-21.8.0
keyrings.alt @ file:///dev/shm/vis.software/Python/3.9.9/system-system/keyringsalt/keyrings.alt-4.1.0
kiwisolver @ file:///dev/shm/vis.software/Python/3.9.9/system-system/kiwisolver/kiwisolver-1.3.2
liac-arff @ file:///dev/shm/vis.software/Python/3.9.9/system-system/liacarff/liac-arff-2.5.0
llvmlite==0.38.0
lockfile @ file:///dev/shm/vis.software/Python/3.9.9/system-system/lockfile/lockfile-0.12.2
lxml @ file:///dev/shm/vis.software/Python/3.9.9/system-system/lxml/lxml-4.6.4
MarkupSafe @ file:///dev/shm/vis.software/Python/3.9.9/system-system/MarkupSafe/MarkupSafe-2.0.1
marshmallow @ file:///dev/shm/vis.software/Python/3.9.9/system-system/marshmallow/marshmallow-3.14.1
marshmallow-polyfield @ file:///dev/shm/vis.software/Python/3.9.9/system-system/marshmallowpolyfield/marshmallow-polyfield-5.10
matplotlib-inline @ file:///dev/shm/vis.software/Python/3.9.9/system-system/matplotlibinline/matplotlib-inline-0.1.3
mistune @ file:///dev/shm/vis.software/Python/3.9.9/system-system/mistune/mistune-0.8.4
mock @ file:///dev/shm/vis.software/Python/3.9.9/system-system/mock/mock-4.0.3
more-itertools @ file:///dev/shm/vis.software/Python/3.9.9/system-system/moreitertools/more-itertools-8.11.0
mpmath @ file:///dev/shm/vis.software/Python/3.9.9/system-system/mpmath/mpmath-1.2.1
msgpack @ file:///dev/shm/vis.software/Python/3.9.9/system-system/msgpack/msgpack-1.0.2
multitasking @ file:///dev/shm/vis.software/Python/3.9.9/system-system/multitasking/multitasking-0.0.10
nbclient @ file:///dev/shm/vis.software/Python/3.9.9/system-system/nbclient/nbclient-0.5.9
nbconvert @ file:///dev/shm/vis.software/Python/3.9.9/system-system/nbconvert/nbconvert-6.3.0
nbformat @ file:///dev/shm/vis.software/Python/3.9.9/system-system/nbformat/nbformat-5.1.3
nbsphinx @ file:///dev/shm/vis.software/Python/3.9.9/system-system/nbsphinx/nbsphinx-0.8.7
nest-asyncio @ file:///dev/shm/vis.software/Python/3.9.9/system-system/nest_asyncio/nest_asyncio-1.5.1
netaddr @ file:///dev/shm/vis.software/Python/3.9.9/system-system/netaddr/netaddr-0.8.0
netifaces @ file:///dev/shm/vis.software/Python/3.9.9/system-system/netifaces/netifaces-0.11.0
networkx @ file:///dev/shm/vis.software/Python/3.9.9/system-system/networkx/networkx-2.6.3
node-semver==0.6.1
nose @ file:///opt/cesga/2020/sources/p/Python/extensions/nose-1.3.7-py3-none-any.whl
notebook @ file:///dev/shm/vis.software/Python/3.9.9/system-system/notebook/notebook-6.4.6
ntlm-auth @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ntlmauth/ntlm-auth-1.5.0
numba==0.55.1
numpy @ file:///dev/shm/vis.software/numpy/1.21.4/system-system-python-3.9.9/numpy-1.21.4
openpyxl @ file:///dev/shm/vis.software/Python/3.9.9/system-system/openpyxl/openpyxl-3.0.9
packaging @ file:///opt/cesga/2020/sources/p/Python/extensions/packaging-20.9-py2.py3-none-any.whl
pandocfilters @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pandocfilters/pandocfilters-1.5.0
paramiko @ file:///dev/shm/vis.software/Python/3.9.9/system-system/paramiko/paramiko-2.7.2
parso @ file:///dev/shm/vis.software/Python/3.9.9/system-system/parso/parso-0.8.2
pastel @ file:///opt/cesga/2020/sources/p/Python/extensions/pastel-0.2.1-py2.py3-none-any.whl
patch-ng==1.17.4
pathlib2 @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pathlib2/pathlib2-2.3.6
paycheck @ file:///dev/shm/vis.software/Python/3.9.9/system-system/paycheck/paycheck-1.0.2
pbr @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pbr/pbr-5.8.0
pexpect @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pexpect/pexpect-4.8.0
pickleshare @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pickleshare/pickleshare-0.7.5
Pillow @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Pillow/Pillow-8.4.0
pkgconfig @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pkgconfig/pkgconfig-1.5.5
pkginfo @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pkginfo/pkginfo-1.8.1
platformdirs @ file:///dev/shm/vis.software/Python/3.9.9/system-system/platformdirs/platformdirs-2.4.0
pluggy @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pluggy/pluggy-0.13.1
pluginbase==1.0.1
ply @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ply/ply-3.11
poetry @ file:///dev/shm/vis.software/Python/3.9.9/system-system/poetry/poetry-1.1.11
poetry-core @ file:///dev/shm/vis.software/Python/3.9.9/system-system/poetrycore/poetry-core-1.0.7
prometheus-client @ file:///dev/shm/vis.software/Python/3.9.9/system-system/prometheus_client/prometheus_client-0.12.0
prompt-toolkit @ file:///dev/shm/vis.software/Python/3.9.9/system-system/prompt_toolkit/prompt_toolkit-3.0.22
psutil @ file:///dev/shm/vis.software/Python/3.9.9/system-system/psutil/psutil-5.8.0
ptyprocess @ file:///opt/cesga/2020/sources/p/Python/extensions/ptyprocess-0.7.0-py2.py3-none-any.whl
py @ file:///dev/shm/vis.software/Python/3.9.9/system-system/py/py-1.11.0
py-expression-eval @ file:///dev/shm/vis.software/Python/3.9.9/system-system/py_expression_eval/py_expression_eval-0.3.14
pyasn1 @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pyasn1/pyasn1-0.4.8
pycairo==1.20.1
pycparser @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pycparser/pycparser-2.20
pycrypto @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pycrypto/pycrypto-2.6.1
Pygments @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Pygments/Pygments-2.10.0
PyGObject==3.42.0
PyJWT==1.7.1
pylatexenc @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pylatexenc/pylatexenc-2.10
pylev @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pylev/pylev-1.4.0
PyNaCl @ file:///dev/shm/vis.software/Python/3.9.9/system-system/PyNaCl/PyNaCl-1.4.0
pyparsing @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pyparsing/pyparsing-2.4.7
pyrsistent @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pyrsistent/pyrsistent-0.18.0
pytest @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pytest/pytest-6.2.5
python-dateutil @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pythondateutil/python-dateutil-2.8.2
pytoml @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pytoml/pytoml-0.1.21
pyts==0.12.0
pytz @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pytz/pytz-2021.3
PyYAML==5.4.1
pyzmq @ file:///dev/shm/vis.software/Python/3.9.9/system-system/pyzmq/pyzmq-22.3.0
regex @ file:///dev/shm/vis.software/Python/3.9.9/system-system/regex/regex-2021.11.10
requests @ file:///dev/shm/vis.software/Python/3.9.9/system-system/requests/requests-2.26.0
requests-ntlm @ file:///opt/cesga/2020/sources/p/Python/extensions/requests_ntlm-1.1.0-py2.py3-none-any.whl
requests-toolbelt @ file:///dev/shm/vis.software/Python/3.9.9/system-system/requeststoolbelt/requests-toolbelt-0.9.1
scandir @ file:///dev/shm/vis.software/Python/3.9.9/system-system/scandir/scandir-1.10.0
scikit-build @ file:///dev/shm/vis.software/Python/3.9.9/system-system/scikitbuild/scikit-build-0.12.0
scikit-learn==1.0.2
scipy==1.8.0
SecretStorage @ file:///dev/shm/vis.software/Python/3.9.9/system-system/SecretStorage/SecretStorage-3.3.1
semantic-version @ file:///dev/shm/vis.software/Python/3.9.9/system-system/semantic_version/semantic_version-2.8.5
Send2Trash @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Send2Trash/Send2Trash-1.8.0
setuptools-rust @ file:///dev/shm/vis.software/Python/3.9.9/system-system/setuptoolsrust/setuptools-rust-1.0.0
setuptools-scm @ file:///dev/shm/vis.software/Python/3.9.9/system-system/setuptools_scm/setuptools_scm-6.3.2
shellingham @ file:///dev/shm/vis.software/Python/3.9.9/system-system/shellingham/shellingham-1.4.0
simplegeneric @ file:///dev/shm/vis.software/Python/3.9.9/system-system/simplegeneric/simplegeneric-0.8.1
simplejson @ file:///dev/shm/vis.software/Python/3.9.9/system-system/simplejson/simplejson-3.17.6
six @ file:///dev/shm/vis.software/Python/3.9.9/system-system/six/six-1.16.0
smmap @ file:///dev/shm/vis.software/Python/3.9.9/system-system/smmap/smmap-5.0.0
snowballstemmer @ file:///dev/shm/vis.software/Python/3.9.9/system-system/snowballstemmer/snowballstemmer-2.2.0
sortedcontainers @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sortedcontainers/sortedcontainers-2.4.0
Sphinx @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Sphinx/Sphinx-4.3.0
sphinx-bootstrap-theme @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxbootstraptheme/sphinx-bootstrap-theme-0.8.0
sphinxcontrib-applehelp @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribapplehelp/sphinxcontrib-applehelp-1.0.2
sphinxcontrib-devhelp @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribdevhelp/sphinxcontrib-devhelp-1.0.2
sphinxcontrib-htmlhelp @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribhtmlhelp/sphinxcontrib-htmlhelp-2.0.0
sphinxcontrib-jsmath @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribjsmath/sphinxcontrib-jsmath-1.0.1
sphinxcontrib-qthelp @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribqthelp/sphinxcontrib-qthelp-1.0.3
sphinxcontrib-serializinghtml @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribserializinghtml/sphinxcontrib-serializinghtml-1.1.5
sphinxcontrib-websupport @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sphinxcontribwebsupport/sphinxcontrib-websupport-1.2.4
svgwrite @ file:///dev/shm/vis.software/Python/3.9.9/system-system/svgwrite/svgwrite-1.4.1
sympy @ file:///dev/shm/vis.software/Python/3.9.9/system-system/sympy/sympy-1.9
tabulate @ file:///dev/shm/vis.software/Python/3.9.9/system-system/tabulate/tabulate-0.8.9
terminado @ file:///dev/shm/vis.software/Python/3.9.9/system-system/terminado/terminado-0.12.1
testpath @ file:///dev/shm/vis.software/Python/3.9.9/system-system/testpath/testpath-0.5.0
threadpoolctl @ file:///dev/shm/vis.software/Python/3.9.9/system-system/threadpoolctl/threadpoolctl-3.0.0
thrift @ file:///dev/shm/vis.software/Python/3.9.9/system-system/thrift/thrift-0.15.0
toml @ file:///dev/shm/vis.software/Python/3.9.9/system-system/toml/toml-0.10.2
tomli @ file:///opt/cesga/2020/sources/p/Python/extensions/tomli-1.2.2-py3-none-any.whl
tomli_w @ file:///opt/cesga/2020/sources/p/Python/extensions/tomli_w-0.4.0-py3-none-any.whl
tomlkit @ file:///opt/cesga/2020/sources/p/Python/extensions/tomlkit-0.7.2-py2.py3-none-any.whl
tornado @ file:///dev/shm/vis.software/Python/3.9.9/system-system/tornado/tornado-6.1
tqdm==4.62.3
traitlets @ file:///dev/shm/vis.software/Python/3.9.9/system-system/traitlets/traitlets-5.1.1
typing_extensions @ file:///opt/cesga/2020/sources/p/Python/extensions/typing_extensions-4.0.0-py3-none-any.whl
ujson @ file:///dev/shm/vis.software/Python/3.9.9/system-system/ujson/ujson-4.3.0
urllib3 @ file:///dev/shm/vis.software/Python/3.9.9/system-system/urllib3/urllib3-1.26.7
versioneer @ file:///dev/shm/vis.software/Python/3.9.9/system-system/versioneer/versioneer-0.21
virtualenv @ file:///dev/shm/vis.software/Python/3.9.9/system-system/virtualenv/virtualenv-20.10.0
Wand @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Wand/Wand-0.6.7
wcwidth @ file:///dev/shm/vis.software/Python/3.9.9/system-system/wcwidth/wcwidth-0.2.5
webencodings @ file:///dev/shm/vis.software/Python/3.9.9/system-system/webencodings/webencodings-0.5.1
websockets @ file:///dev/shm/vis.software/Python/3.9.9/system-system/websockets/websockets-10.1
Werkzeug @ file:///dev/shm/vis.software/Python/3.9.9/system-system/Werkzeug/Werkzeug-2.0.2
widgetsnbextension @ file:///dev/shm/vis.software/Python/3.9.9/system-system/widgetsnbextension/widgetsnbextension-3.5.1
xlrd @ file:///dev/shm/vis.software/Python/3.9.9/system-system/xlrd/xlrd-2.0.1
XlsxWriter @ file:///dev/shm/vis.software/Python/3.9.9/system-system/XlsxWriter/XlsxWriter-3.0.2
xlutils @ file:///dev/shm/vis.software/Python/3.9.9/system-system/xlutils/xlutils-2.0.0
xlwt @ file:///dev/shm/vis.software/Python/3.9.9/system-system/xlwt/xlwt-1.3.0
zipfile36 @ file:///dev/shm/vis.software/Python/3.9.9/system-system/zipfile36/zipfile36-0.1.3
zipp @ file:///dev/shm/vis.software/Python/3.9.9/system-system/zipp/zipp-3.6.0
zope.interface @ file:///dev/shm/vis.software/Python/3.9.9/system-system/zopeinterface/zope.interface-5.4.0

I'm going step by step: I run deeplexicon and get this error:

python3 deeplexicon.py -p /fast5_pass/ -f multi -m  models/pAmps-final-actrun_newdata_nanopore_UResNet20v2_model.030.h5 > output.tsv
Traceback (most recent call last):
  File "/mnt/lustre/scratch/nlsas/home/csic/mgr/ogr/programs/deeplexicon/deeplexicon.py", line 21, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

then I try to install tensorflow but got this issue (even though I installed numba 0.55.1 and numpy 1.20 previously):

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
numba 0.55.1 requires numpy<1.22,>=1.18, but you have numpy 1.22.3 which is incompatible.
Successfully installed absl-py-1.0.0 astunparse-1.6.3 cachetools-5.0.0 certifi-2021.10.8 charset-normalizer-2.0.12 flatbuffers-2.0 gast-0.5.3 google-auth-2.6.2 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.44.0 h5py-3.6.0 idna-3.3 importlib-metadata-4.11.3 keras-2.8.0 keras-preprocessing-1.1.2 libclang-13.0.0 markdown-3.3.6 numpy-1.22.3 oauthlib-3.2.0 opt-einsum-3.3.0 protobuf-3.19.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-2.27.1 requests-oauthlib-1.3.1 rsa-4.8 setuptools-61.2.0 six-1.16.0 tensorboard-2.8.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.8.0 tensorflow-io-gcs-filesystem-0.24.0 termcolor-1.1.0 tf-estimator-nightly-2.8.0.dev2021122109 typing-extensions-4.1.1 urllib3-1.26.9 werkzeug-2.1.0 wheel-0.37.1 wrapt-1.14.0 zipp-3.7.0
Psy-Fer commented 2 years ago

Okay try downgrading numpy by removing it and installing 1.20 again. It probably grabbed a higher version when doing something else.

Then do numba install again.

By any chance can you run docker on your HPC?

Psy-Fer commented 2 years ago

Oh and try these instructions for vbz

https://github.com/nanoporetech/vbz_compression/tree/master/python/pyvbz

ogrecio commented 2 years ago

Done. We have the udocker: udocker/1.1.3-python-2.7.15 module available but I have never used it before.

Now, the problem comes from h5py (even though I installed it):

pip install h5py -t $STORE/.conda

Collecting h5py
  Downloading h5py-3.6.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 17.1 MB/s eta 0:00:00
Collecting numpy>=1.14.5
  Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 234.1 MB/s eta 0:00:00
Collecting cached-property
  Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Installing collected packages: cached-property, numpy, h5py
Successfully installed cached-property-1.5.2 h5py-3.6.0 numpy-1.21.5
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/h5py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/__pycache__ already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/cached_property-1.5.2.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/numpy-1.21.5.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/numpy already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/cached_property.py already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/numpy.libs already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/h5py-3.6.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/h5py.libs already exists. Specify --upgrade to force replacement.
WARNING: Target directory /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/bin already exists. Specify --upgrade to force replacement.
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$
(deeplexicon_env) [csmgrogr@login210-19 deeplexicon]$ python3 deeplexicon.py -p /mnt/lustre/scratch/nlsas/home/csic/mgr/ogr/SarsCov2/directRNA/test1/20220323_1439_X1_FAS46772_0b6ed64a/fast5_pass/ -f multi -m  models/pAmps-final-actrun_newdata_nanopore_UResNet20v2_model.030.h5 > output.tsv
Traceback (most recent call last):
  File "deeplexicon.py", line 14, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'

For the .whl file, I got these errors (I tried different .whl files):

 pip install https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.0/pyvbz-1.0.0-cp35-cp35m-linux_x86_64.whl -t $STORE/.conda
ERROR: pyvbz-1.0.0-cp35-cp35m-linux_x86_64.whl is not a supported wheel on this platform.
pip install https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.0/pyvbz-1.0.0-cp35-cp35m-linux_x86_64.whl -t $STORE/.conda^C
 pip install https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.1/pyvbz-1.0.1-cp38-cp38-linux_x86_64.whl -t $STORE/.conda
ERROR: pyvbz-1.0.1-cp38-cp38-linux_x86_64.whl is not a supported wheel on this platform.
pip install https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.1/pyvbz-1.0.1-cp37-cp37-linux_x86_64.whl -t $STORE/.conda
ERROR: pyvbz-1.0.1-cp37-cp37-linux_x86_64.whl is not a supported wheel on this platform.
pip install https://github.com/nanoporetech/vbz_compression/releases/download/v1.0.1/pyvbz-1.0.1-cp38-cp38-linux_aarch64.whl -t $STORE/.conda
ERROR: pyvbz-1.0.1-cp38-cp38-linux_aarch64.whl is not a supported wheel on this platform.
ogrecio commented 2 years ago

Hi James, the IT team helped installing the program in a conda environment. Then we are having this error now:

python3 deeplexicon.py -p /fast5_pass/ -f multi -m models/pAmps-final-actrun_newdata_nanopore_UResNet20v2_model.030.h5 > output.tsv Using TensorFlow backend. Traceback (most recent call last): File "deeplexicon.py", line 624, in main() File "deeplexicon.py", line 229, in main model = read_model(config[deeplexicon][trained_model]) if args.config else read_model(args.model) File "deeplexicon.py", line 134, in read_model model = load_model(model_name) # as a path File "/mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/keras/engine/saving.py", line 419, in load_model model = _deserialize_model(f, custom_objects, compile) File "/mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/keras/engine/saving.py", line 224, in _deserialize_model model_config = json.loads(model_config.decode('utf-8')) AttributeError: 'str' object has no attribute 'decode'

Does it sound like something you could provide any help? Thanks!

Psy-Fer commented 2 years ago

Hey,

So i think that is an issue with h5py. See this issue reply for a potential fix:

https://github.com/Psy-Fer/deeplexicon/issues/13#issuecomment-876567110

ogrecio commented 2 years ago

Hi James, I got a new issue:

 python3 deeplexicon.py -p /mnt/lustre/scratch/nlsas/home/csic/mgr/ogr/SarsCov2/directRNA/test1/20220323_1439_X1_FAS46772_0b6ed64a/fast5_pass/ -f multi -m  models/pAmps-final-actrun_newdata_nanopore_UResNet20v2_model.030.h5 > output.tsv
Using TensorFlow backend.
WARNING:tensorflow:From /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2022-04-07 18:54:05.871205: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2022-04-07 18:54:05.885194: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2022-04-07 18:54:05.885318: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55df204c8580 executing computations on platform Host. Devices:
2022-04-07 18:54:05.885336: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
  File "deeplexicon.py", line 624, in <module>
    main()
  File "deeplexicon.py", line 229, in main
    model = read_model(config[deeplexicon][trained_model]) if args.config else read_model(args.model)
  File "deeplexicon.py", line 137, in read_model
    metrics=['accuracy'])
  File "/mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/keras/engine/training.py", line 96, in compile
    self.optimizer = optimizers.get(optimizer)
  File "/mnt/netapp1/Store_CSIC/home/csic/mgr/ogr/.conda/envs/deeplexicon/lib/python3.7/site-packages/keras/optimizers.py", line 801, in get
    str(identifier))
ValueError: Could not interpret optimizer identifier: <tensorflow.python.keras.optimizers.Adam object at 0x149ad9316790>

Thanks for your help

Psy-Fer commented 2 years ago

Umm, i'm not sure what that error means, I've not seen that error before.

I'll have to have a look into it.

lpryszcz commented 6 months ago

I was trying to add vbz support to deeplexicon, but if you install newer version of https://github.com/nanoporetech/ont_fast5_api it updated h5py, which results in errors like in #13

the only viable solution is to change compression of your Fast5 to gzip compress_fast5 --compression gzip or use https://github.com/biocorecrg/MoP3