search

rsemeraro / PyPore

A python tool box for fast and accurate quality control, conversion and alignment of nanopore sequencing data
GNU General Public License v3.0
20 stars 5 forks source link

ImportError: No module named _multiarray_umath #3

Closed fpaskali closed 5 years ago

fpaskali commented 5 years ago

I am trying to use PyPore on Manjaro Linux 18.0.4 I installed all dependencies normally, except hdf5-dev, which I had to install from .deb package.

When I use pypore seqstats -i test_folder/test_dataset/fail -l label

I get this error [PyPore][INFO] - Start Main Parser ImportError: No module named _multiarray_umath Traceback (most recent call last): File "/usr/bin/pypore", line 11, in load_entry_point('PyPore==0.1', 'console_scripts', 'pypore')() File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 61, in main run_subtool(mod, ags, parser) File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 16, in run_subtool sq.run(ags) File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 125, in run ArgsReader(args) File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 52, in ArgsReader sq.run(ags) File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 1106, in run fast5module(work_dir, file_folder, prefix, tmp_dir, Fast_flag, size) File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 905, in fast5module import fast5_reader as f5r File "/usr/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/fast5_reader.py", line 9, in import h5py File "/usr/lib/python2.7/site-packages/h5py/init.py", line 36, in from ._conv import register_converters as _register_converters File "h5py/h5t.pxd", line 14, in init h5py._conv File "h5py/h5t.pyx", line 1, in init h5py.h5t ImportError: numpy.core.multiarray failed to import

I think it has something to do with numpy version, currently using 1.16.0

rsemeraro commented 5 years ago

Hi, i've never worked with Manjaro distribution, anyway, could you try to update your numpy version and try again? I agree with you, it is something related to numpy.
Alternatively, could you create conda environments on your system?

fpaskali commented 5 years ago

I tried with conda environment, made python 2.7 with all dependencies and I am getting the same error. What other distribution do you recommend to try?

rsemeraro commented 5 years ago

try with the 1.15.0...

fpaskali commented 5 years ago

Same. I will try now on debian based distribution

rsemeraro commented 5 years ago

Hi, some good news? :)

fpaskali commented 5 years ago

I tried with MXLinux(Debian 9) and Conda2, it works with test data, but I was getting empty values report. And with mine data I was getting some other error.

Now I tried on Ubuntu 19.04. It works perfectly with test_data, but with my data I get this error.

[PyPore][INFO] - Start Main Parser Traceback (most recent call last): File "/home/filip/anaconda2/bin/pypore", line 11, in load_entry_point('PyPore==0.1', 'console_scripts', 'pypore')() File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 61, in main run_subtool(mod, ags, parser) File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 16, in run_subtool sq.run(ags) File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 125, in run ArgsReader(args) File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 52, in ArgsReader sq.run(ags) File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 1106, in run fast5module(work_dir, file_folder, prefix, tmp_dir, Fast_flag, size) File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 909, in fast5module for e,x in enumerate(dir_surfer(file_folder)): File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 883, in dir_surfer if en == 0 and int(os.path.basename(subdir)) == 0 and multiread_flag == False: ValueError: invalid literal for int() with base 10: 'hfu_stim_data'

rsemeraro commented 5 years ago

Ok, it seems a "folder architecture error", we could fix it in a short time. Could show me the data folder structure? Maybe, by moving data we fix the problem in a bit.

fpaskali commented 5 years ago

This is the folder structure.

Stim_Act
└── 20190516_0838_MN23412_FAK77219_16eff389
    ├── fast5_fail
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_0.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_1.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_2.fast5
    │   └── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_3.fast5
    ├── fast5_pass
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_0.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_10.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_11.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_12.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_13.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_14.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_15.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_16.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_17.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_1.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_2.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_3.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_4.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_5.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_6.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_7.fast5
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_8.fast5
    │   └── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_9.fast5
    ├── fast5_skip
    │   └── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_0.fast5
    ├── fastq_fail
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_0.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_1.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_2.fastq
    │   └── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_3.fastq
    ├── fastq_pass
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_0.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_10.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_11.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_12.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_13.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_14.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_15.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_16.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_17.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_1.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_2.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_3.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_4.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_5.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_6.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_7.fastq
    │   ├── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_8.fastq
    │   └── FAK77219_fb8de8d4935a650d9154dd8b32cc4649c6a2882b_9.fastq
    ├── final_summary.txt
    └── sequencing_summary
        └── mlsnb7_243_20190516_103821_FAK77219_minion_sequencing_run_Stim_Act_sequencing_summary.txt

I tried to do something similar like test_dataset, by moving fast5_pass and fast5_fail folders into new directory and renaming them to pass and fail, as well for fastq_pass/fail.

rsemeraro commented 5 years ago

Ok, i think that by moving fast5_fail and fast5_pass in a empty directory, such as, fast5, in order to obtain a structure like:

Stim_Act
└── 20190516_0838_MN23412_FAK77219_16eff389
    ├── fast5
           ├── fast5_pass
           ├── fast5_fail

Will fix the problem. The path to use as input will be: /path/to/Stim_Act/20190516_0838_MN23412_FAK77219_16eff389/fast5

fpaskali commented 5 years ago

pypore seqstats -i hfu_folder/fast5 -l HFU_data -fq yes -n 4

[PyPore][INFO] - Start Main Parser
Traceback (most recent call last):
  File "/home/filip/anaconda2/bin/pypore", line 11, in <module>
    load_entry_point('PyPore==0.1', 'console_scripts', 'pypore')()
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 61, in main
    run_subtool(mod, ags, parser)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/pypore.py", line 16, in run_subtool
    sq.run(ags)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 125, in run
    ArgsReader(args)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seqstats.py", line 52, in ArgsReader
    sq.run(ags)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 1106, in run
    fast5module(work_dir, file_folder, prefix, tmp_dir, Fast_flag, size)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 909, in fast5module
    for e,x in enumerate(dir_surfer(file_folder)):
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 894, in dir_surfer
    min_time_catcher(f)
  File "/home/filip/anaconda2/lib/python2.7/site-packages/PyPore-0.1-py2.7.egg/lib/seq_routines.py", line 824, in min_time_catcher
    StartTime = f5['UniqueGlobalKey/tracking_id'].attrs['exp_start_time'].decode('UTF-8')
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/filip/anaconda2/lib/python2.7/site-packages/h5py/_hl/group.py", line 262, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: 'Unable to open object (component not found)'
rsemeraro commented 5 years ago

Ok, try to point only the pass folder... -i hfu_folder/fast5/fast5_pass

fpaskali commented 5 years ago

Same error

rsemeraro commented 5 years ago

Ok, it would be a problem if I ask you to give me a single fast5?

fpaskali commented 5 years ago

Sent to your mail

rsemeraro commented 5 years ago

Ok, you are dealing with multi_read fast5. Try to activate the --multi_read_fast5 flag by typing --multi_read_fast5 yes

pypore seqstats -i hfu_folder/fast5 -l HFU_data -fq yes -n 4 -m yes
fpaskali commented 5 years ago

Its working. Thank you very much. There is small problem. The PyPore is not terminating itself after Plotting, just hanging on [PyPore][INFO] - Finish. I waited for 10 min and terminated it manually.

rsemeraro commented 5 years ago

ok! i'll check for this issue and i'll fix it! Thank you for your suggestion!

rsemeraro commented 5 years ago

Fixed!

fpaskali commented 5 years ago

I retried in Manjaro and it works great now. I guess it was a folder tree structure problem.

rsemeraro commented 5 years ago

Cool! :)