search

metageni / SUPER-FOCUS

A tool for agile functional analysis of shotgun metagenomic data
GNU General Public License v3.0
21 stars 12 forks source link

Database error: BLAST and Diamond #41

Closed s4251484 closed 5 years ago

s4251484 commented 5 years ago

Hi

I installed SUPER-FOCUS using miniconda today. The database_PKs.txt and organisms2subsystem.txt were not in the db folder, which I manually downloaded and put into the db folder.

ls /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db 100_clusters 98_clusters 90_clusters clusters organisms2subsystem.txt 95_clusters database_PKs.txt

Here are the two database errors I encountered:

1. BLAST BLAST Database error: No alias or index file found for protein database [/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/static/blast/90.db] in search path

[2019-03-12 15:57:29,724 - INFO] Parsing Alignments Traceback (most recent call last): File "/Users/ME/miniconda2/envs/super-focus/bin/superfocus", line 11, in load_entry_point('superfocus==0.31', 'console_scripts', 'superfocus')() File "/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/superfocus.py", line 322, in main minimum_identity, minimum_alignment, subsystems_translation, aligner) File "/Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/do_alignment.py", line 171, in parse_alignments update_results(results, sample_index, temp_results, normalise, number_samples) UnboundLocalError: local variable 'temp_results' referenced before assignment

2. DIAMOND

CPU threads: 4

Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1) Temporary directory: /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/tmp Opening the database... No such file or directory [0.000117s] Error: Error opening file /Users/ME/miniconda2/envs/super-focus/lib/python3.7/site-packages/superfocus_app/db/static/diamond/90.db

I reckon there is something not right with my db folder, but I don't know how to make it right. Please help. Looking forward for your reply. Thanks.

metageni commented 5 years ago

Hi @s4251484 , Any reason to be using SUPER-FOCUS 0.31 rather than the latest?

Also, did you can the program to download the database (https://github.com/metageni/SUPER-FOCUS#database)?

Best

s4251484 commented 5 years ago

Hi @s4251484 , Any reason to be using SUPER-FOCUS 0.31 rather than the latest?

Also, did you can the program to download the database (https://github.com/metageni/SUPER-FOCUS#database)?

Best

hi @metageni, 0.31 was the version I got when I downloaded SUPER-FOCUS via miniconda. and it was not updated even after I "conda update super-focus" and yes I downloaded both blast and diamond databases.

UPDATE I: I install SUPER-FOCUS using pip3 intall instead and got the latest version 0.33. however it didn't manage to format the diamond database successfully, my guess is there is problem with the makedb. it doesn't seem to be able to read and iterate the individual faa files and make a db out of the faa files. Hope my experience helps you.

Other the other hand, downloading and formatting blast database (with super-focus intalled via pip3) was successful, I'm now running the superfocus with blast database, hopefully comes back with good news.

UPDATE II: both (pip3) and (miniconda) installed super-focus are not able to build diamond db for me. errors came out after unzipping the db.zip. hence, I went on to make the diamond db manually (by referring to diamond manual) instead of using the superfocus_downloadDB.py. Briefly, first use cat to join the multiple faa files into one fasta file, then use diamond makedb to format fasta into diamond db. Move the diamond db files to a folder "db/static/diamond". then the diamond *.db files will be usable with superfocus.py.

PS: dont forget to download the database_PKs.txt and organisms2subsystems.txt from github and transfer them to the db folder if yours are not auto-generated. this happened to me.

Cheers!

metageni commented 5 years ago

@s4251484

I just took a look into the problem, and it seems that version 0.33 would fix it. For some reason, bioconda is installing v0.31.

I just downloaded version 0.33 on git and installed it. I downloaded the database and everything worked fine.

For now, I would recommend you do the same.

I will follow up with the bioconda team on the issue.

Sorry about it

s4251484 commented 5 years ago

@metageni that will be great! thank you~ thank you for developing useful tools like this. They are great for bio students who dont have a programming background, eg ME!

fconstancias commented 5 years ago

I am experiencing the same issue running superfocus installed using conda.

metageni commented 5 years ago

Hi @fconstancias, I will see if this weekend I can create a new version that deals with the database installation in a different way. Stay tuned!

metageni commented 5 years ago

@fconstancias @s4251484 I have released a new version (v0.34) of superfocus (https://github.com/metageni/SUPER-FOCUS/releases) where users can now manually download the database and format it using a script I wrote.

It should be live on bioconda in 1-2 days. I will see if I can push it tomorrow into pip.

For now, you can pip it using (https://github.com/metageni/SUPER-FOCUS/releases).

Please read (https://github.com/metageni/SUPER-FOCUS#database) on how to download and format the database.

Any feedback is welcome.

ghost commented 5 years ago

I have tried installing v0.34 but at the end, it is installing 0.0.0 version.

[root@rameshgoel ~]# cd /srv/data/SUPER-FOCUS-0.34/ [root@rameshgoel SUPER-FOCUS-0.34]# python setup.py install /usr/bin/miniconda3/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'use_scm_version' warnings.warn(msg) running install running bdist_egg running egg_info creating superfocus.egg-info writing superfocus.egg-info/PKG-INFO writing dependency_links to superfocus.egg-info/dependency_links.txt writing entry points to superfocus.egg-info/entry_points.txt writing requirements to superfocus.egg-info/requires.txt writing top-level names to superfocus.egg-info/top_level.txt writing manifest file 'superfocus.egg-info/SOURCES.txt' reading manifest file 'superfocus.egg-info/SOURCES.txt' writing manifest file 'superfocus.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib creating build/lib/superfocus_app copying superfocus_app/superfocus_downloadDB.py -> build/lib/superfocus_app copying superfocus_app/init.py -> build/lib/superfocus_app copying superfocus_app/do_alignment.py -> build/lib/superfocus_app copying superfocus_app/superfocus.py -> build/lib/superfocus_app creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/superfocus_app copying build/lib/superfocus_app/superfocus_downloadDB.py -> build/bdist.linux-x86_64/egg/superfocus_app copying build/lib/superfocus_app/init.py -> build/bdist.linux-x86_64/egg/superfocus_app copying build/lib/superfocus_app/do_alignment.py -> build/bdist.linux-x86_64/egg/superfocus_app copying build/lib/superfocus_app/superfocus.py -> build/bdist.linux-x86_64/egg/superfocus_app byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/superfocus_downloadDB.py to superfocus_downloadDB.cpython-36.pyc byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/init.py to init.cpython-36.pyc byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/do_alignment.py to do_alignment.cpython-36.pyc byte-compiling build/bdist.linux-x86_64/egg/superfocus_app/superfocus.py to superfocus.cpython-36.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying superfocus.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO creating dist creating 'dist/superfocus-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing superfocus-0.0.0-py3.6.egg removing '/usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg' (and everything under it) creating /usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg Extracting superfocus-0.0.0-py3.6.egg to /usr/bin/miniconda3/lib/python3.6/site-packages superfocus 0.0.0 is already the active version in easy-install.pth Installing superfocus script to /usr/bin/miniconda3/bin Installing superfocus_downloadDB script to /usr/bin/miniconda3/bin

Installed /usr/bin/miniconda3/lib/python3.6/site-packages/superfocus-0.0.0-py3.6.egg Processing dependencies for superfocus==0.0.0 Searching for scipy==1.2.1 Best match: scipy 1.2.1 Processing scipy-1.2.1-py3.6-linux-x86_64.egg scipy 1.2.1 is already the active version in easy-install.pth

Using /usr/bin/miniconda3/lib/python3.6/site-packages/scipy-1.2.1-py3.6-linux-x86_64.egg Searching for numpy==1.16.2 Best match: numpy 1.16.2 Processing numpy-1.16.2-py3.6-linux-x86_64.egg numpy 1.16.2 is already the active version in easy-install.pth Installing f2py script to /usr/bin/miniconda3/bin Installing f2py3 script to /usr/bin/miniconda3/bin Installing f2py3.6 script to /usr/bin/miniconda3/bin

Using /usr/bin/miniconda3/lib/python3.6/site-packages/numpy-1.16.2-py3.6-linux-x86_64.egg Finished processing dependencies for superfocus==0.0.0

[root@rameshgoel SUPER-FOCUS-0.34]# superfocus -v SUPER-FOCUS 0.0.0

Why it's not installing v0.34?

metageni commented 5 years ago 
Genis-MacBook-Pro-2:geni.silva$ superfocus -v
SUPER-FOCUS 0.34

Bizzare! I wonder if it is something to do with bioconda.

I will need to investigate it

ghost commented 5 years ago

Hi @metageni : I was able to install v0.34 using super user by logging as root. But I can't access the superfocus when log in as user. How to fix this?

metageni commented 5 years ago

Problem solved here https://github.com/metageni/SUPER-FOCUS/issues/49

metageni commented 5 years ago

Closing this issue, and leaving (https://github.com/metageni/SUPER-FOCUS/issues/24) so people can ask questions there.

fconstancias commented 5 years ago

I have just installed super focus using conda conda create -n super-focus -y -c bioconda super-focus and I also got

(super-focus)$ superfocus --version SUPER-FOCUS 0.0.0

metageni commented 5 years ago

@fconstancias For some reason, superfocus is giving this back when using -v. Were you able to run the program? I will add a fix to it on next release.

You can also install the too with pip.