search

biobakery / biobakery_workflows

bioBakery workflows is a collection of workflows and tasks for executing common microbial community analyses using standardized, validated tools and parameters.
http://huttenhower.sph.harvard.edu/biobakery_workflows
Other
102 stars 33 forks source link

Cannot install full shotgun databases (wmgx) in the biobakery VM image #6

Open hyfood opened 3 years ago

hyfood commented 3 years ago

Dear developer(s),

I am new to the biobakery VM images and currently I am going through the biobakery_workflows tutorials following:

https://github.com/biobakery/biobakery/wiki/biobakery_workflows

However, I found it difficult to install the full shotgun databases following the "1.2.5 Install databases" in the tutorial. Below is my command and the returns as shown in the terminal:

**vagrant@biobakery:~$ biobakery_workflows_databases --install wmgx Installing humann utility mapping database Download URL: http://huttenhower.sph.harvard.edu/humann2_data/full_mapping_v201901.tar.gz Downloading file of size: 2.55 GB

2.55 GB 100.00 % 9.49 MB/sec 0 min -0 sec
Extracting: /home/vagrant/biobakery_workflows_databases/humann/full_mapping_v201901.tar.gz

Database installed: /home/vagrant/biobakery_workflows_databases/humann/utility_mapping

Unable to write to the HUMAnN config file. Unable to install database. Error running command: humann_databases --download utility_mapping full /home/vagrant/biobakery_workflows_databases/humann**

May be something is wrong with the HUMAnN? I tried re-downloading the humann but it didn't work. May you please advise how to address this issue?

Thanks in advance

Warm regards

Hong

ljmciver commented 3 years ago

Hi @HongYao4523685 , Thank you for the detailed post and sorry for any confusion. It looks like the database might have been successfully installed at /home/vagrant/biobakery_workflows_databases/humann/utility_mapping but there was an error writing to the config file. With our new vagrant provisioning configuration where we install our packages with pip we use sudo for the install. So since the tool was installed as root an edit to the configuration file will also need to be run as root. If you run the same command again in the VM but run it with "sudo" it should resolve the config write error you are seeing.

Thank you, Lauren

hyfood commented 3 years ago

Hi Lauren,

Thank you so much!

I tried "sudo biobakery_workflows_databases --install wmgx" and the humann configuration problem has been solved. However, Another error comes. Please see the below for the details:

image

It seems like it is something to do with the bowtie index...

Thanks again in advance!

Warm regards

Hong

ljmciver commented 3 years ago

Hi Hong, Glad to hear that solved the error! The workflow database install expects the metaphlan database to be installed prior. To install this database run $ metaphlan --install. Once this database is installed it should resolve the errors you are seeing.

Thank you, Lauren

hyfood commented 3 years ago

Hi Lauren,

Thanks! I run $ sudo metaphlan --install prior to the sudo biobakery_workflows_databases --install wmgx, and the bowtie index problem has been solved. Then several databases were downloaded and extracted, but another error comes:

image

It seems like it is unable to install the database again.

Could you please advise how to address this issue?

Thanks heaps for your patience Hong

hyfood commented 3 years ago

Hi Lauren

Finally, the database is successfully installed! The problem is that the biobakery VM image only has the disk size of 64 GB. After I extended it to 256 GB I got no errors!

Thanks for your helps again!!

Hong

hyfood commented 3 years ago

Sorry to disturb again.....

Then, I tried the next step in the tutorial: biobakery_workflows wmgx --input input --output output_data --bypass-strain-profiling, using the samples provided in the tutorial

Some tasks completed, but more tasks failed: image

And from the error report, the reason is that the metaphlan_databases directory is not wrieable: image

I tried added the sudo prior to the command again but the results are the same.

Could you please suggest how to ideal with this issue?

Warm regards

Hong

ljmciver commented 3 years ago

Hello Hong,

That is great you figured out the disk space issue! The latest version of metaphlan will not check the database installed with a method that requires write permissions. If you update your metaphlan install to the latest version it should resolve the errors you are seeing. Alternatively you can change the permissions on the folder that contains the metaphlan databases.

Thank you, Lauren

hyfood commented 3 years ago

Hi Lauren,

Thanks for the instructions!

I update the latest metaphlan version and change the permissions as well, and the task 16 was successful. Then all the other tasks run well, but the task 21 which failed:

task21

And the error information is: task21-info

Any advise to address this issue?

Thanks again in advance!

Kind regards

Hong

ljmciver commented 3 years ago

Hello @HongYao4523685 , Sorry for my slow response! I have been a bit swamped with a upcoming software deadline. Is it possible that the humann v3 source is no longer installed or that the $PATH or $PYTHONPATH have changed?

Thank you, Lauren

hyfood commented 3 years ago

Hi Lauren,

Thanks for your response and it is very exciting to hear that your new software is coming!

No need to worry about my issue as I successfully went through the Kneaddata, Metaphlan3 and Humann3 separately in my computer. After coming across some problems, I realize that the problems may come from both the software&database versions and path.

I am grateful for your patience and helps!

Warm regards

Hong