search

micom-dev / q2-micom

A Qiime plugin for MICOM.
https://micom-dev.github.io/q2-micom
Apache License 2.0
10 stars 4 forks source link

error with qiime micom build and qiime micom grow commands #1

Closed Mechah closed 4 years ago

Mechah commented 4 years ago

Dear Sean and Christian,

I'm very curious about micom and I tested it as a new plugin for qiime2. However, I experienced several issues:

First I tried to apply the tool on my own example dataset, but received the following error with the qiime micom build command: Plugin error from micom: Length mismatch: Expected axis has 15 elements, new values have 8 elements

Ok then I thought I should stick to your example data covered in your tutorial (https://github.com/micom-dev/q2-micom/blob/master/docs/README.md) However, for the crc_table.qza and crc_taxa.qza files I received the following error with qiime micom build: (1/2) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2 Artifact (.qza) (2/2) Invalid value for "--i-taxonomy": 'crc_taxa.qza' is not a QIIME 2 Artifact (.qza)

Hm, then I thought let's try the small dataset of the moving picture tutorial (https://docs.qiime2.org/2019.10/tutorials/moving-pictures/) and the qiime micom build command worked! :slight_smile:

Nevertheless, with the qiime micom grow command I got another error: Plugin error from micom: 'NoneType' object is not subscriptable I guess this was due to several warnings I received during the execution that qiime micom grow could not solve cooperative tradeoffs for many samples.

So I was wondering how to proceed from here. Would be great if you could comment on those errors or maybe check the artifacts of the crc files, or recommend other suitable test datasets to get an idea how to get micom work...

By the way I used qiime2 2019.10 and the gurobi license.

Thanks for any support!

cdiener commented 4 years ago

Oh no, I'm sorry it didn't work as expected I will investigate and get back to you. I might need some time to check what is going on with Gurobi since I don't have access to it at the moment but maybe it related to something else.

cdiener commented 4 years ago

Hey I tried all the samples you mentioned with cplex and Qiime 2 2019.10 and it all went fine. However, as mentioned in the forum it was a bit of work installing the package here since I had to upgrade a lot of dependencies by hand and needed to install pyarrow with pip due to a conda conflict. So I would recommend using the dev environment until the new Qiime 2 version is released (scheduled for this month). I also encountered errors with the visualizations and this is now fixed in version 0.5.1.

qiime micom grow --i-models models.qza \
                 --i-medium western_diet_gut.qza \
                 --p-tradeoff 0.5 \
                 --p-threads 4 \
                 --o-results growth.qza \
                 --verbose
/home/cdiener/miniconda3/envs/qiime2-2019.10/lib/python3.6/site-packages/skbio/util/_testing.py:15: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as pdt
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34/34 [18:51<00:00, 33.28s/sample(s)]
Saved MicomResults to: growth.qza

The crossover errors are probably due to some problem with Gurobi. I'll try to get my hand on it and investigate more. For now the my recommendation would be to use CPLEX. I will update the tutorial to mention those pitfalls.

Planned changes

Mechah commented 4 years ago

Dear Christian,

thanks for your support. I updated to version 0.5.1 and installed CPLEX. However, I still get the same errors. Since I now have both Gurobi and CPLEX installed as my QP solver - is there an easy way to force micom to use CPLEX over Gurobi? I also remember that there were several issues when I installed micom the first time. Maybe I did not solve all issues back then correctly? For now I guess I'll wait for the next qiime2 release (2020.2) and then give micom a new try...

cdiener commented 4 years ago

Hmm not in the current version of q2-micom. I will update micom to prefer cplex. If you installed q2-micom in its own environment you could simply uninstall Gurobi there. Your local Gurobi license will not be touched by this and remain intact. If you don't want to wait you can also create a developer environment for Qiime 2 which is pretty much the new version at this point. You can find instructions for this here: https://dev.qiime2.org/latest/quickstart/.

said3427 commented 4 years ago

I´m having the same problem using Gurobi and CPLEX. Using the tutorial data I got warnings in all samples, it took 2 hours using 16 cores:

2020-02-28 18:09:24.628 | WARNING | q2_micom._growth:_growth:36 - Could not solve cooperative tradeoff for Cancer7

After that I got the NoneType error:

Traceback (most recent call last): File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/q2cli/commands.py", line 328, in call results = action(arguments) File "</home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/decorator.py:decorator-gen-243>", line 2, in grow File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable output_types, provenance) File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callableexecutor
output_views = self._callable(view_args) File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/q2_micom/_growth.py", line 66, in grow growth = pd.concat(r["growth"] for r in results) File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 255, in concat sort=sort, File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/pandas/core/reshape/concat.py", line 301, in init objs = list(objs) File "/home/said/anaconda3/envs/qiime2-dev/lib/python3.6/site-packages/q2_micom/_growth.py", line 66, in growth = pd.concat(r["growth"] for r in results) TypeError: 'NoneType' object is not subscriptable

Plugin error from micom:

'NoneType' object is not subscriptable

See above for debug info.

cdiener commented 4 years ago

Yes, I saw the same thing. There was a bug in micom where it would not use Gurobi when installed. However that would only affect environments where cplex is not available. This is fixed now in micom so you could try just to update micom.

What is the output of 

import cplex
print(cplex.__version__)

when run in Python of your Qiime 2 environment?

said3427 commented 4 years ago

The cplex version: 12.10.0.0

Mechah commented 4 years ago

Hi Christian, I got the following error when I tried to install micom 0.5.1 in a qiime2 2020.2 environment:

Successfully built q2-micom Installing collected packages: tqdm, micom, q2-micom, python-libsbml-experimental Attempting uninstall: tqdm Found existing installation: tqdm 4.7.2 ERROR: Cannot remove entries from nonexistent file /home/cluster/o_mahnerta/miniconda3/envs/qiime2-2020.2/lib/python3.6/site-packages/easy-install.pth

Thanks for your help!

cdiener commented 4 years ago

Yes, that is something I need to circumvent in the install. The problem is that bioconda and conda-forge both have tqdm and bioconda has a very outdated one. Bioconda has the priority in the current installation instructions which is wrong. You can try to fix it with conda install -c conda-forge "tqdm>=4.14".

cdiener commented 4 years ago

Okay, I updated the installation instructions in the tutorial and Github Readme. Could you try with those in a fresh Qiime environment? It worked for me on my Mac, but would be good to see if it does for you.

said3427 commented 4 years ago

I updated qiime micom, I have both CPLEX ('12.10.0.0') and Gurobi (9.0.1) installed.

I still have the error:

2020-03-02 21:51:26.312 | ERROR | q2_micom._growth:_growth:24 - Community models were not built with a QP-capable solver. This means that you did not install CPLEX or Gurobi. If you did install one of the two please file a bug report at https://github.com/micom-dev/q2-micom/issues.

and

Plugin error from micom:

All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in qiime micom build to create simpler models.

See above for debug info.

I am working in Centos 7.4

cdiener commented 4 years ago

Did you run qiime micom build again after updating? This is necessary to build the models with the correct solver.

said3427 commented 4 years ago

It works for me 🎉

cdiener commented 4 years ago

This seems to be fixed. Feel free to reopen if the problems persist with the new package versions and installation instructions.

Mechah commented 4 years ago

Hi, I tried the new version micom 0.5.4 in a new qiime2-2020.2 environment with CPLEX. However, I experienced the same issues again.

Using the tutorial dataset (crc_table.qza, crc_taxa.qza) with the micom build command:

(1/2) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2 Artifact (.qza) (2/2) Invalid value for "--i-taxonomy": 'crc_taxa.qza' is not a QIIME 2 Artifact (.qza)

Using the moving pictures tutorial dataset with the micom grow command:

Plugin error from micom:

All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in qiime micom build to create simpler models.

and finally with my own dataset, again with micom build:

Plugin error from micom:

Length mismatch: Expected axis has 15 elements, new values have 8 elements

Sorry for bothering you again...

cdiener commented 4 years ago

Hi, similar as in the comment before you will need to rerun qiime micom build for all data sets so the new changes can take effect. You get the optimization failures because this hasn't happened. Since the tutorial data works for others it should be something specific to your setup.

How did you download the data? Could you try to download it again making sure to use the "Download" button in the upper right corner? I have already seen those error pop up in broken Qiime 2 environments since an error in the plugin system often yields those as well. Does this also occur in a fresh Qiime 2 environment? Does running qiime dev refresh-cache give any output?

The error on your own data does look like a bug. Unfortunately, it's hard to me to say what is going on without seeing the artifacts. How were those tables generated and what classifier did you use to assign taxonomy? Could you paste the first few lines of your taxonomy data?

Mechah commented 4 years ago

Hi, I got the crc table and taxa via the wget command. Now, I did it as you suggested (using the download button in github) and it seems to work (currently processing).

Regarding my own data: They were created with a qiime2 versions from 2018 and 2019 using DADA2 for denoising. If you think there might be a compatibility issue with older qiime2 artifacts I could dig deeper into it. Taxonomy was assigned with a classifier based on Silva132 with OTUs clustered at 99% and trimmed to V4 515f-806r primer regions. Here are the first lines of my taxonomy file:

Feature ID Taxon Confidence fd879d3291c346b2d51ba6debeaf88d1 D_0Bacteria;D_1__Proteobacteria;D_2Betaproteobacteria;D_3Burkholderiales;D_4__Comamonadaceae;D_5Variovorax 0.7325667438214208 f5252d4b7dd1919e759c5b58925e8012 D_0Bacteria;D_1__Proteobacteria;D_2Betaproteobacteria;D_3Rhodocyclales;D_4__Rhodocyclaceae;D_5Methyloversatilis;D_6uncultured bacterium;D_7__;D_8;D_9;D_10__;D_11;D_12;D_13__;D_14 0.802512653515797 c62e8ab3f6ad5aa1e30472076ac2011b D_0Bacteria;D_1__Cyanobacteria;D_2Chloroplast 0.9999545292928743

thanks for your help!

cdiener commented 4 years ago

That actually looks good. I also used SILVA for the tutorial data, so that should work.

Did you really cluster the OTUs after DADA2? This is usually not necessary since DADA2 is in itself kind of a (better) clustering algorithm for 16S data. But this should not be issue here.

What I can see from your taxa is that it is likely not a human gut sample and there are many more taxa ranks than I would expect. When I use the SILVA classifier I never get more than 7 taxa ranks (kingdom, phylum, class, order, family, genus, species). So this is surprising to me:

D_0__Bacteria;D_1__Proteobacteria;D_2__Betaproteobacteria;D_3__Rhodocyclales;D_4__Rhodocyclaceae;D_5__Methyloversatilis;D_6__uncultured bacterium;D_7__;D_8__;D_9__;D_10__;D_11__;D_12__;D_13__;D_14__

That is likely where the error comes from since that taxon has 15 ranks and q2-micom assumes at most 8. I will investigate a bit how to fix that cleanly but can probably add a dirty fix that just uses up to 8 ranks and gives a warning. I'll get on that...

I should note that most of your taxa are probably not contained in AGORA which is pretty specific to the human gut. I am in the process of providing the CarveME model DB (>5K genera) for q2-micom as well but this will still take a little bit.

Mechah commented 4 years ago

Hi, I did not cluster after denoising with DADA2, but the Silva classifier is based on clustered representative sequences... The 15 levels might be a result of a different classifier using 515f-926r primers, which are also targeting some Eukaryota. I'm a bit surprised that I see these levels here. Have to double check...

Next, I will use a different test dataset (due to the gut specific model database) and make sure I'll only have 7 levels in my taxonomy artifact.

Unfortunately, I had another error after successfully building the model with micom grow and the crc tutorial dataset:

Plugin error from micom:

All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in qiime micom build to create simpler models.

Looking forward to get my hands on the CarveME model DB as soon as it is available and the above errors could be fixed...

cdiener commented 4 years ago

I did not cluster after denoising with DADA2, but the Silva classifier is based on clustered representative sequences... The 15 levels might be a result of a different classifier using 515f-926r primers, which are also targeting some Eukaryota. I'm a bit surprised that I see these levels here. Have to double check...

Okay this makes more sense.

Unfortunately, I had another error after successfully building the model with micom grow and the crc tutorial dataset:

Plugin error from micom:

All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in qiime micom build to create simpler models.

This is odd. If you run this with the --verbose option do you get a warning that there is no QP-capable solver? You can check your Gurobi installation with something like the following:

python -c "import gurobipy; gurobipy.Model()"
gurobi_cl --version

and your cplex installation with:

python -c "import cplex; print(cplex.__version__)"
cdiener commented 4 years ago

I asked on the Qiime 2 Forum and they recommend to use a revised SILVA classifier that will only give 6-7 level taxonomies -> https://forum.qiime2.org/t/silva-138-for-qiime2/12957. This would probably fix your issue with your own data set.

Mechah commented 4 years ago

Hi, sorry for my late response (a bit difficult to find time for testing due to corona virus restrictions...).

Thanks for flagging the post on silva-138! I re-classified my sequences and now I can make models with all the tested datasets (no more length mismatch errors) :-)

However with micom grow I still get the following error (irrespective if it is the tutorial data or my own data). I also tested my cplex installation with python -c "import cplex; print(cplex.version)" and then I get this: 12.9.0.0 So cplex seems to be there. (I did not install gurobi in this environment so with the other command I receive an error... - not very much surprising).

I assume there is a problem with cplex and micom here (maybe specifically in my conda environment), despite I followed each step in your installation tutorial and did not get any errors...

If you have any ideas, I'm happy to keep on testing...

cdiener commented 4 years ago

This is pretty odd. I have tried it with that cplex version and the the tutorial data and it works in my environment. It could be a problem with the environment setup or with the model building. which OS are you running on?

Mechah commented 4 years ago

Hi, I use the following OS on our cluster: Linux sx036 5.4.0-0.bpo.3-amd64 #1 SMP Debian 5.4.13-1~bpo10+1 (2020-02-07) x86_64 And q2-micom runs in qiime2-2020.2 in a conda environment with the following version: 4.8.2 (conda/4.8.2 requests/2.22.0 CPython/3.7.3 Linux/5.4.0-0.bpo.3-amd64 debian/10 glibc/2.28)

cdiener commented 4 years ago

Bit at a loss here since I also use Debian and have similar versions. Would you mind providing the commands/scripts you are running? Or did you paste the tutorial commands verbatim?

Mechah commented 4 years ago

Yes, more or less. Here are my commands and once again the error message...

qiime micom build --i-abundance table-dada2.qza --i-taxonomy taxonomy.qza --i-models /home/cluster/o_mahnerta/databases/micom/agora_genus_103.qza --p-cutoff 0.0001 --p-threads 12 --o-community-models models.qza --verbose

qiime micom grow --i-models models.qza --i-medium /home/cluster/o_mahnerta/databases/micom/western_diet_gut.qza --p-tradeoff 0.3 --p-threads 48 --o-results growth.qza --verbose

Plugin error from micom:

All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in qiime micom build to create simpler models.

cdiener commented 4 years ago

That looks good. Do you think you could share your models.qza file for the tutorial data with me? That way I could check if something looks amiss.

Mechah commented 4 years ago

Sorry the file is too big to put it here. You can download it from this folder here: https://box.medunigraz.at/s/3Lcstmy93QBA9P4

cdiener commented 4 years ago

So I ran qiime micom grow on your built models and it ran without a hitch so I would suspect it may be related to your environment. Here is what I ran:

qiime micom grow --i-models ~/Downloads/crc_models.qza \                                     
                 --i-medium western_diet_gut.qza \
                 --p-tradeoff 0.3 \
                 --p-threads 4 \
                 --o-results crc_growth.qza \
                 --verbose

As a last resort cold you also try running the command with --p-threads 16. There are only 16 samples so the plugin will not parallelize beyond that. Also make sure you have about 1GB of RAM per core.

If that does not work this will unfortunately be hard to investigate but a good initial check would be to run some of the MICOM test data in Python to see if there are some problems. Also make sure you did install a full license for cplex (can be academic) and not the community edition provided on conda (limits the number of variables). For instance you could run the following in your Qiime python:

import micom
tax = micom.data.test_taxonomy()
com = micom.Community(tax)
sol = com.cooperative_tradeoff(fraction=1.0)
print(sol.members)
Mechah commented 4 years ago

Hi, after a long time I got back to test micom again. And you were right - my issue was related to the community edition of cplex. After installing the academic full license qiime micom grow worked out finally. Thanks for all your help!

cdiener commented 4 years ago

Awesome. Glad that worked. I should make a note in the installation instructions that the community edition won't work.