search

jts / nanocorrect

Experimental pipeline for correcting nanopore reads
MIT License
39 stars 10 forks source link

pysam version issue? #5

Closed 4learning closed 9 years ago

4learning commented 9 years ago

When run "python nanocorrect.py" I got the following error: Traceback (most recent call last): File "nanocorrect.py", line 187, in (seq, n_reads) = run_poa_and_consensus(overlaps, read_idx) File "nanocorrect.py", line 146, in run_poa_and_consensus (in_fn, n_reads) = write_poa_input(overlaps, read_idx) File "nanocorrect.py", line 62, in write_poa_input read_id1 = ref.references[read_idx] AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'

python 2.7.3 pysam: 0.7.5

jts commented 9 years ago

Can you provide the complete set of commands that you ran?

4learning commented 9 years ago

Thanks.

Here are the commands tested (within the installation path of nanocorrect): make -f nanocorrect-overlap.make INPUT=reads.fasta NAME=nc python nanocorrect.py nc > corrected.fasta

where reads.fasta are minion data (pass filter).

jts commented 9 years ago

You need to provide a range of reads to correct:

python nanocorrect.py nc 1:50 > corrected.fasta

This will correct the first 50 reads in the file.

4learning commented 9 years ago

Sorry. I tried different range setting. and got the same error.

Many thanks in advance

4learning commented 9 years ago

nc.db: files = 1 411 nc.pp m000000_000000_00000_c1898213712391273 blocks = 1 size = 50 cutoff = 0 all = 0 0 0 411 411

python nanocorrect.py nc 2:20 > corrected.fasta Traceback (most recent call last): File "nanocorrect.py", line 187, in (seq, n_reads) = run_poa_and_consensus(overlaps, read_idx) File "nanocorrect.py", line 146, in run_poa_and_consensus (in_fn, n_reads) = write_poa_input(overlaps, read_idx) File "nanocorrect.py", line 62, in write_poa_input read_id1 = ref.references[read_idx] AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'

jts commented 9 years ago

What version of pysam do you have?

jts commented 9 years ago

Sorry I see the version above now

jts commented 9 years ago

Can you first try running:

samtools faidx reads.pp.fasta

If that doesn't work, try using the latest pysam from pip (0.8.3) in a virtualenv?

4learning commented 9 years ago

the samtools faidx produced an output like this: m000000_000000_00000_c1898213712391273/0/0_8253 8253 156 60 61 m000000_000000_00000_c1898213712391273/1/0_8020 8020 8700 60 61 m000000_000000_00000_c1898213712391273/2/0_8041 8041 17005 60 61 m000000_000000_00000_c1898213712391273/3/0_1310 1310 25337 60 61 m000000_000000_00000_c1898213712391273/4/0_1329 1329 26822 60 61 m000000_000000_00000_c1898213712391273/5/0_1287 1287 28325 60 61 m000000_000000_00000_c1898213712391273/6/0_3448 3448 29790 60 61 m000000_000000_00000_c1898213712391273/7/0_2970 2970 33449 60 61 m000000_000000_00000_c1898213712391273/8/0_3544 3544 36620 60 61 m000000_000000_00000_c1898213712391273/9/0_3480 3480 40382 60 61 m000000_000000_00000_c1898213712391273/10/0_3302 3302 44076 60 61 m000000_000000_00000_c1898213712391273/11/0_3496 3496 47588 60 61 m000000_000000_00000_c1898213712391273/12/0_8995 8995 51302 60 61 m000000_000000_00000_c1898213712391273/13/0_8970 8970 60603 60 61 m000000_000000_00000_c1898213712391273/14/0_8880 8880 69877 60 61 m000000_000000_00000_c1898213712391273/15/0_530 530 79061 60 61 m000000_000000_00000_c1898213712391273/16/0_558 558 79753 60 61 m000000_000000_00000_c1898213712391273/17/0_542 542 80472 60 61 m000000_000000_00000_c1898213712391273/18/0_12754 12754 81182 60 61 m000000_000000_00000_c1898213712391273/19/0_11851 11851 94304 60 61 m000000_000000_00000_c1898213712391273/20/0_12923 12923 106506 60 61 m000000_000000_00000_c1898213712391273/21/0_4495 4495 119802 60 61 m000000_000000_00000_c1898213712391273/22/0_4638 4638 124526 60 61 m000000_000000_00000_c1898213712391273/23/0_4346 4346 129394 60 61 m000000_000000_00000_c1898213712391273/24/0_2795 2795 133970 60 61 m000000_000000_00000_c1898213712391273/25/0_2727 2727 136966 60 61 m000000_000000_00000_c1898213712391273/26/0_2749 2749 139891 60 61 m000000_000000_00000_c1898213712391273/27/0_7956 7956 142843 60 61 m000000_000000_00000_c1898213712391273/28/0_7953 7953 151086 60 61 m000000_000000_00000_c1898213712391273/29/0_7737 7737 159324 60 61 m000000_000000_00000_c1898213712391273/30/0_8246 8246 167347 60 61 m000000_000000_00000_c1898213712391273/31/0_7415 7415 175885 60 61 m000000_000000_00000_c1898213712391273/32/0_8279 8279 183576 60 61 m000000_000000_00000_c1898213712391273/33/0_3384 3384 192150 60 61 m000000_000000_00000_c1898213712391273/34/0_3041 3041 195745 60 61 m000000_000000_00000_c1898213712391273/35/0_3495 3495 198989 60 61 m000000_000000_00000_c1898213712391273/36/0_17091 17091 202703 60 61 m000000_000000_00000_c1898213712391273/37/0_16829 16829 220236 60 61 m000000_000000_00000_c1898213712391273/38/0_16862 16862 237501 60 61 m000000_000000_00000_c1898213712391273/39/0_6059 6059 254804 60 61 m000000_000000_00000_c1898213712391273/40/0_5665 5665 261120 60 61 m000000_000000_00000_c1898213712391273/41/0_5946 5946 267034 60 61 m000000_000000_00000_c1898213712391273/42/0_14352 14352 273238 60 61 m000000_000000_00000_c1898213712391273/43/0_13474 13474 287985 60 61 m000000_000000_00000_c1898213712391273/44/0_14036 14036 301837 60 61 m000000_000000_00000_c1898213712391273/45/0_7977 7977 316264 60 61 m000000_000000_00000_c1898213712391273/46/0_7887 7887 324528 60 61 m000000_000000_00000_c1898213712391273/47/0_7857 7857 332699 60 61 m000000_000000_00000_c1898213712391273/48/0_7969 7969 340846 60 61 m000000_000000_00000_c1898213712391273/49/0_7607 7607 349104 60 61 m000000_000000_00000_c1898213712391273/50/0_8002 8002 356992 60 61 m000000_000000_00000_c1898213712391273/51/0_3500 3500 365285 60 61 m000000_000000_00000_c1898213712391273/52/0_3616 3616 368998 60 61 m000000_000000_00000_c1898213712391273/53/0_3441 3441 372827 60 61 m000000_000000_00000_c1898213712391273/54/0_6208 6208 376483 60 61 m000000_000000_00000_c1898213712391273/55/0_6069 6069 382949 60 61 m000000_000000_00000_c1898213712391273/56/0_6074 6074 389272 60 61 m000000_000000_00000_c1898213712391273/57/0_485 485 395604 60 61 m000000_000000_00000_c1898213712391273/58/0_422 422 396251 60 61 m000000_000000_00000_c1898213712391273/59/0_550 550 396832 60 61 m000000_000000_00000_c1898213712391273/60/0_3485 3485 397551 60 61 m000000_000000_00000_c1898213712391273/61/0_3455 3455 401251 60 61 ....

jts commented 9 years ago

That output is in the file reads.pp.fasta.fai, right?

4learning commented 9 years ago

yes :-)

jts commented 9 years ago

Did you have a chance to try pysam 0.8.3?

4learning commented 9 years ago

Sorry for my late response ...

I tried to set up a virtual machine within nanocorrect installation dir. What I did was the following: virtualenv --no-site-packages test_vm source test_vm/bin/activate pip install pysam source test_vm/bin/activate

which python /misc/ngseq7/src/nanocorrect/test_vm/bin/python

more test_vm/lib/python2.7/site-packages/pysam/version.py

pysam versioning information

version = "0.8.3"

samtools_version = "1.2"

htslib_version = "1.2.1"

but still the same error showed up when nanocorrect was run: (test_vm)$ make -f nanocorrect-overlap.make INPUT=reads.fasta NAME=nc3 nanocorrect-preprocess.pl reads.fasta > nc3.pp.fasta fasta2DB nc3 nc3.pp.fasta DBsplit -s50 nc3 DBdust nc3 HPCdaligner -t5 -mdust nc3 > HPCcommands.txt /bin/bash HPCcommands.txt LAcat nc3 > nc3.las rm nc3.*.las (test_vm)$ python nanocorrect.py nc3 1:411 > corrected.fasta Traceback (most recent call last): File "nanocorrect.py", line 187, in (seq, n_reads) = run_poa_and_consensus(overlaps, read_idx) File "nanocorrect.py", line 146, in run_poa_and_consensus (in_fn, n_reads) = write_poa_input(overlaps, read_idx) File "nanocorrect.py", line 62, in write_poa_input read_id1 = ref.references[read_idx] AttributeError: 'pysam.csamtools.Fastafile' object has no attribute 'references'

Many thanks.

jts commented 9 years ago

I haven't forgotten about this but I'm a bit stumped. @alexjironkin had a similar problem today but after upgrading to 0.8.3 he reports that nanocorrect runs properly

alexjironkin commented 9 years ago

In python console if you do:

import pysam
pysam.__version__

Do you also get 0.8.3?

4learning commented 9 years ago

Thanks, Jared and Alex.

Alex you are right and it is also what has puzzled me. In python console (started within the VM), the version of pysam is still 0.7.5. You know the fix to this? Sorry that I am not very familiar with Python

alexjironkin commented 9 years ago

Hmm, that's really strange. It looks like there are 2 versions of pysam on your system.

To see where it is do the following in Python console:

import pysam, os
print os.path.abspath(pysam.__file__)

This should show you absolute path to the pyc file with pysam 0.7.5. Depending where it is you may be able to uninstall it using

pip uninstall pysam

What OS/Python is this?

4learning commented 9 years ago

I am using Debian GNU/Linux 7.8 (wheezy). It is a shared server with multi-user access.

I found the path to pysam 0.7.5 following your instructions. But when I did pip uninstall pysam, either within or outside VM, the path did not show up in the list (as shown below):

/usr/local/ngseq/src/nanocorrect$ source test_vm/bin/activate (test_vm):/usr/local/ngseq/src/nanocorrect$ python Python 2.7.3 (default, Mar 13 2014, 11:03:55) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import pysam, os print os.path.abspath(pysam.file) /usr/local/ngseq/lib/python/pysam/init.pyc quit() (test_vm):/usr/local/ngseq/src/nanocorrect$ deactivate /usr/local/ngseq/src/nanocorrect$ pip uninstall pysam Uninstalling pysam: /usr/local/lib/python2.7/dist-packages/pysam /usr/local/lib/python2.7/dist-packages/pysam-0.8.0.egg-info Proceed (y/n)? n /usr/local/ngseq/src/nanocorrect$ source test_vm/bin/activate (test_vm):/usr/local/ngseq/src/nanocorrect$ pip uninstall pysam Uninstalling pysam: /misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam /misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam-0.8.3-py2.7.egg-info Proceed (y/n)? n

alexjironkin commented 9 years ago

What's the output of the following in Python console both for virtualenv and none:

import os, sys

print os.environ['PYTHONPATH']
print sys.path
4learning commented 9 years ago

within VM:

print os.environ['PYTHONPATH'] :/usr/local/ngseq/lib/python/:/usr/local/ngseq/lib/python2.7:/usr/local/ngseq/lib/python2.7/site-packages:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules:/usr/local/ngseq/stow/qiime-r1487/lib/:/usr/local/ngseq/stow/PyNAST-1.1/lib/:/usr/local/ngseq/src/Denoiser_0.851/:/usr/local/ngseq/lib/python:/usr/local/ngseq/lib/python2.6/site-packages print sys.path ['', '/usr/local/ngseq/lib/python/keyring-3.3-py2.7.egg', '/usr/local/ngseq/lib/python/requests-2.1.0-py2.7.egg', '/usr/local/ngseq/lib/python/distribute-0.6.28-py2.7.egg', '/usr/local/ngseq/lib/python/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg', '/usr/local/ngseq/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/distribute-0.6.24-py2.7.egg', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/misc/ngseq7/src/nanocorrect', '/usr/local/ngseq/lib/python', '/usr/local/ngseq/lib/python2.7', '/usr/local/ngseq/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/plat-linux2', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-tk', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-old', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/misc/ngseq7/src/nanocorrect/test_vm/local/lib/python2.7/site-packages', '/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages']

outside VM:

print os.environ['PYTHONPATH'] :/usr/local/ngseq/lib/python/:/usr/local/ngseq/lib/python2.7:/usr/local/ngseq/lib/python2.7/site-packages:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar:/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules:/usr/local/ngseq/stow/qiime-r1487/lib/:/usr/local/ngseq/stow/PyNAST-1.1/lib/:/usr/local/ngseq/src/Denoiser_0.851/:/usr/local/ngseq/lib/python:/usr/local/ngseq/lib/python2.6/site-packages print sys.path ['', '/usr/local/ngseq/lib/python/keyring-3.3-py2.7.egg', '/usr/local/ngseq/lib/python/requests-2.1.0-py2.7.egg', '/usr/local/ngseq/lib/python/distribute-0.6.28-py2.7.egg', '/usr/local/ngseq/lib/python/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg', '/usr/local/ngseq/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/h5py-2.3.1-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/poretools-0.3.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/rpy2-2.4.3-py2.7-linux-x86_64.egg', '/misc/ngseq7/src/nanocorrect', '/usr/local/ngseq/lib/python', '/usr/local/ngseq/lib/python2.7', '/usr/local/ngseq/lib/python2.7/site-packages', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar', '/usr/local/ngseq/src/modil_beta/structural_variations/mrfstructvar/modules', '/usr/local/ngseq/stow/qiime-r1487/lib', '/usr/local/ngseq/stow/PyNAST-1.1/lib', '/usr/local/ngseq/src/Denoiser_0.851', '/usr/local/ngseq/lib/python2.6/site-packages']

Have a nice weekend

alexjironkin commented 9 years ago

It looks like your '/usr/local/ngseq/lib/python2.7' is before '/misc/ngseq7/src/nanocorrect/test_vm/lib'

I don't know how that happened. If there is no system requirement to have pysam 8.0 I would suggest upgrading to 8.3.

As a quick hack puting

PYTHONPATH=/misc/ngseq7/src/nanocorrect/test_vm/lib/python2.7/site-packages/pysam:$PYTHONPATH python blah blah blah

inside your VM should temporarily change the search paths and your pysam will appear firstin the search list.It is a little mystery how the PYTHONPATH got tangled.

jts commented 9 years ago

As this seems to be a python install issue rather than a nanocorrect problem I'm closing for now. Please reopen it if persists after fixing your paths as suggested by @alexjironkin.