search

macmanes-lab / Oyster_River_Protocol

Official Repository of the Oyster River Protocol for Transcriptome Assembly
Creative Commons Zero v1.0 Universal
16 stars 14 forks source link

PySlice_Unpack error #16

Open cooketho opened 5 years ago

cooketho commented 5 years ago

oyster.mk main fails on the test data set with the following error. Quick google search reveals similar errors in conda, sometimes reported fixed by using newer Python.

======= SPAdes pipeline finished.

SPAdes log can be found here: /home/tc/sampledata/assemblies/test.spades_k55/spades.log

Thank you for using SPAdes! Traceback (most recent call last): File "/lab/pr/tc/ORP/software/transabyss/transabyss", line 18, in from utilities.adj_utils import has_edges File "/lab/pr/tc/ORP/software/transabyss/utilities/adj_utils.py", line 6, in import igraph File "/lab/pr/tc/ORP/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in from igraph._igraph import * ImportError: /lab/pr/tc/ORP/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefine\ d symbol: PySlice_Unpack make: *** [/home/tc/sampledata/assemblies/test.transabyss.fasta] Error 1

cooketho commented 5 years ago

Fixed it by editing py36_env.yml to specify a newer python version (>=3.6.1), adding git to dependencies, and getting busco directly from the gitlab repository (v3.1) instead of the out-of-date anaconda repository (v3.02). Don't know if this latter bit was necessary, but I saw in the commits that there were problems with busco when using python>3.6. Anyway, it didn't give me any problems on the included test data. Here is my new py36_env.yml:

name: orp_v2
channels:
  - bioconda
  - conda-forge
  - defaults
  - r
  - antoined
dependencies:
  - python>=3.6.1
  - git
  - zsh
  - ruby
  - make
  - glib
  - zlib==1.2.11=0
  - libgcc>=7.2.0
  - salmon>=0.10.2
  - seqtk
  - trinity
  - spades
  - blast
  - hmmer
  - samtools
  - blat
  - parallel
  - last
  - rcorrector==1.0.3
  - mcl
  - abyss>=2.0.1
  - python-igraph
  - diamond
  - cd-hit
  - bwa
  - pip:
    - biopython
    - cvxopt
    - cython
    - numpy
    - pandas
    - pip
    - scipy
    - bashplotlib
    - "git+https://gitlab.com/ezlab/busco"
macmanes commented 5 years ago

So just to confirm, the test dataset now runs properly, and gives an output something like 

 6| #
 5| ##
 4| ###
 3| ###
 2| ###       #
 1| ### ###   #
   -----------

------------------------------------
|             Summary              |
------------------------------------
|         observations: 20         |
|       min value: -1.000000       |
|         mean : -0.986450         |
|       max value: -0.935000       |
------------------------------------

*****  See the following link for interpretation *****
*****  https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html *****

*****  QUALITY REPORT FOR: test4 using the ORP version 2.1.0 ****
*****  THE ASSEMBLY CAN BE FOUND HERE: /mnt/lustre/macmaneslab/macmanes/fastq_files/assemblies/test4.ORP.fasta ****

*****  BUSCO SCORE ~~~~~>               C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
*****  TRANSRATE SCORE ~~~~~>           0.42269
*****  TRANSRATE OPTIMAL SCORE ~~~~~>   0.57644
*****  UNIQUE GENES ORP ~~~~~>          39
*****  UNIQUE GENES TRINITY ~~~~~>      31
*****  UNIQUE GENES SPADES55 ~~~~~>     22
*****  UNIQUE GENES SPADES75 ~~~~~>     25
*****  UNIQUE GENES TRANSABYSS ~~~~~>   36
cooketho commented 5 years ago

Here is the output:

[bam_sort_core] merging from 0 files and 10 in-memory blocks...
-parsing file: test.sorted.bam
-done parsing file, examining orientations of reads.

 7|  #         
 6| ##         
 5| ##         
 4| ##         
 3| ###        
 2| ###       #
 1| ###  ##   #
   -----------

------------------------
|       Summary        |
------------------------
|   observations: 20   |
| min value: -1.000000 |
|   mean : -0.987400   |
| max value: -0.935000 |
------------------------

*****  See the following link for interpretation ***** 
*****  https://oyster-river-protocol.readthedocs.io/en/latest/strandexamine.html ***** 

*****  QUALITY REPORT FOR: test using the ORP version 2.1.0 ****
*****  THE ASSEMBLY CAN BE FOUND HERE: /home/tc/sampledata2/assemblies/test.ORP.fasta **** 

*****  BUSCO SCORE ~~~~~>               C:0.0%[S:0.0%,D:0.0%],F:0.3%,M:99.7%,n:303
*****  TRANSRATE SCORE ~~~~~>           0.41937
*****  TRANSRATE OPTIMAL SCORE ~~~~~>   0.58487
*****  UNIQUE GENES ORP ~~~~~>          39
*****  UNIQUE GENES TRINITY ~~~~~>      31
*****  UNIQUE GENES SPADES55 ~~~~~>     22
*****  UNIQUE GENES SPADES75 ~~~~~>     23
*****  UNIQUE GENES TRANSABYSS ~~~~~>   35

DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
macmanes commented 5 years ago

ok good.. the other issues aside, have you been able to successfully run a real dataset through the pipeline?

AntonioBaeza commented 5 years ago

Sample problem not solved following the instructions above. This is the way py36_env.yml looks after editing with nano (note:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml name: orp_v2 channels:

The error message I get is the same than before editing py36 but slightly different to the one reported above by cooketho:

Thank you for using SPAdes! Traceback (most recent call last): File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in from utilities.adj_utils import has_edges File "/home/ant/Oyster_River_Protocol/software/transabyss/utilities/adj_utils.py", line 6, in import igraph File "/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in from igraph._igraph import * ImportError: /home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_Unpack make: *** [/home/ant/Oyster_River_Protocol/sampledata/assemblies/test.transabyss.fasta] Error 1 (orp_v2) [ant@hillary sampledata]$

Any thoughts? Thanks beforehand

cooketho commented 5 years ago

You have python==3.6.0. You probably need python>=3.6.1, which is what I used (see previous post).

AntonioBaeza commented 5 years ago

Yes, I did try it too: I edited again py36 so that it looks like:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cat py36_env.yml name: orp_v2 channels:

But I get the same error:

Thank you for using SPAdes! Traceback (most recent call last): File "/home/ant/Oyster_River_Protocol/software/transabyss/transabyss", line 18, in from utilities.adj_utils import has_edges File "/home/ant/Oyster_River_Protocol/software/transabyss/utilities/adj_utils.py", line 6, in import igraph File "/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/init.py", line 34, in from igraph._igraph import * ImportError: /home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_Unpack make: *** [/home/ant/Oyster_River_Protocol/sampledata/assemblies/test.transabyss.fasta] Error 1 (orp_v2) [ant@hillary sampledata]$

Thanks!

cooketho commented 5 years ago

You still have - busco in the main dependencies list. That will install the outdated anaconda repository for busco. Try deleting it. That's why I used the gitlab busco repository installed via pip. You can just cut and paste the code from my previous post.

AntonioBaeza commented 5 years ago

got it! I will try

AntonioBaeza commented 5 years ago

Nope. Still the same error! Not sure what else to do. Thanks

cooketho commented 5 years ago

Hmm. Sorry! I don't have any more advice, except you might want to try deleting the entire installation and starting from scratch by downloading from github again, and following the installation instructions. But before you enter make, copy-paste my post above into your py36_env.yml

AntonioBaeza commented 5 years ago

ok, great, will do, thanks again

macmanes commented 5 years ago

I've updated the repo with the Python changes. You should be able to git update -> make at this point.

Let me know if starting from scratch works - I'm committed to helping you get this up and running.

AntonioBaeza commented 5 years ago

Thanks Matt. I will keep you posted. Regards!

AntonioBaeza commented 5 years ago

I uninstalled everything, and all went smoothly. Now, before doing make, I modified the py36 file, but then I am getting this error message after make:

Collecting git+https://gitlab.com/ezlab/busco (from -r /home/ant/Oyster_River_Protocol/condaenv._g9hpm_0.requirements.txt (line 9)) Cloning https://gitlab.com/ezlab/busco to /tmp/pip-req-build-kzyqy7sh error: RPC failed; result=22, HTTP code = 404

This problem was solved by adding '.git' at the end of the web address in the py36 file

AntonioBaeza commented 5 years ago

The installation went smoothly: I modified the py36 file according to the post above before 'make' and all run well. The previous error was resolved. However, I have a new issue. When I test the protocol, this is the message:

(orp_v2) [ant@hillary Oyster_River_Protocol]$ cd sampledata/

(orp_v2) [ant@hillary sampledata]$ /home/ant/Oyster_River_Protocol/oyster.mk main \

STRAND=RF \ MEM=15 \ CPU=8 \ READ1=test.1.fq.gz \ READ2=test.2.fq.gz \ RUNOUT=test which: no run_BUSCO.py in which: no run_BUSCO.py in (/home/ant/Oyster_River_Protocol/software/anaconda/install/envs/orp_v2/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/condabin:/usr/local/rvm/gems/ruby-1.9.3-p551/bin:/usr/local/rvm/gems/ruby-1.9.3-p551@global/bin:/usr/local/rvm/rubies/ruby-1.9.3-p551/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/rvm/bin:/home/ant/bin:/home/ant/Oyster_River_Protocol/software/anaconda/install/bin:/home/ant/Oyster_River_Protocol/software/OrthoFinder/orthofinder:/home/ant/Oyster_River_Protocol/software/orp-transrate:/home/ant/Oyster_River_Protocol/software/transabyss) /home/ant/Oyster_River_Protocol/oyster.mk:96: "\n\n BUSCO is not installed, must fix ***". Stop. (orp_v2) [ant@hillary sampledata]$

So, I went to github and downloaded the script 'run_BUSCO.py' to anaconda/bin When I run the test sample, the protocol is getting stuck in orthofuse with the following message:

parallel: Error: --tollef has been retired. parallel: Error: Remove --tollef or use --gnu to override --tollef. make: *** [/home/ant/Oyster_River_Protocol/sampledata/orthofuse/test/orthotransrate.done] Error 255 (orp_v2) [ant@hillary sampledata]$

trying to solve this issue now: problem solved by adding flag --gnu to each parallel command found in each *.mk file.

AntonioBaeza commented 5 years ago

I wonder about what can be done if you are working on a linux centos machine with no administrator privileges. I got stuck because the version of coreutils in the machine I am using is old enough so that the command sort does not have --parallel. I installed a new version of coreutils with conda in the ORP_v2 environment [together with other mandatory updates requested by conda when installing coreutils]. Of course, this last action [mandatory script updates] broke the pipeline. I assume that is too much work to rework the pipeline so to have coreutils be integral part of it? Any comment will be appreciated. Thanks

macmanes commented 5 years ago

I will address ASAP! #