search

biocore / mmvec

Neural networks for microbe-metabolite interaction analysis
BSD 3-Clause "New" or "Revised" License
117 stars 51 forks source link

Error when running q2-mmvec paired-omics: 'rU' referenced before assignment #103

Closed ADGM closed 4 years ago

ADGM commented 4 years ago

Hi,

Running mmvec paired-omics in qiime2-2019.7 is throwing this error:

Traceback (most recent call last):
  File "$HOME/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py", line 327, in __call__
    results = action(**arguments)
  File "<$HOME/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-525>", line 2, in paired_omics
  File "$HOME.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 240, in bound_callable
    output_types, provenance)
  File "$HOME/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py", line 383, in _callable_executor_
    output_views = self._callable(**view_args)
  File "$HOME/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/mmvec/q2/_method.py", line 57, in paired_omics
    loss, cv = model.fit(epoch=epochs, summary_interval=summary_interval)
  File "$HOME/.conda/envs/qiime2-2019.7/lib/python3.6/site-packages/mmvec/multimodal.py", line 274, in fit
    self.U = rU
UnboundLocalError: local variable 'rU' referenced before assignment

Hope I can be helped with this; I can't proceed further.

Angeli

mortonjt commented 4 years ago

weird. something must have gone wrong during the install.

How did you install mmvec? What version of mmvec did you install? The sure-fire way to get mmvec installed within a qiime2 conda environment is through the q2stats-workshop tutorial. Specifically

wget https://raw.githubusercontent.com/mortonjt/q2stats-workshops/master/environment.yml
conda env create -n q2mmvec-env -f environment.yml

Let me know if this works.

ADGM commented 4 years ago

Hi, may I know if the environment installs everything from scratch including qiime2 or just mmvec ?

mortonjt commented 4 years ago

It installs everything in a new environment, including qiime2-2019.7 (so that everything is in one command)

On Mon, Nov 18, 2019, 10:57 AM ADGM notifications@github.com wrote:

Hi, may I know if the environment installs everything from scratch including qiime2 or just mmvec ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocore/mmvec/issues/103?email_source=notifications&email_token=AA75VXPIRXTA65YVODOBDM3QUK3NBA5CNFSM4JORYTUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEK5PEY#issuecomment-555079571, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXLNEATT4DNQJIGKQITQUK3NBANCNFSM4JORYTUA .

ADGM commented 4 years ago

I tried the suggested fix but the issue persists, within qiime or on its own :'(

I encountered the following error messages during install:

` SafetyError: The package for r-base located at $HOME/.conda/pkgs/r-base-3.5.1-h08e1455_1008 appears to be corrupted. The path 'lib/R/doc/html/packages.html' has a sha256 mismatch. reported sha256: 7d66db1bf6965741f31d52ee0085f61fac29983b5ba6d5372785531af39c0299 actual sha256: 79ac35eb16acac57cbdd9452bcd8976940baa3f80af562093e065a09d38d4b9c

ClobberError: This transaction has incompatible packages due to a shared path. packages: bioconda::perl-compress-raw-bzip2-2.086-pl526hf484d3e_0, bioconda::perl-digest-hmac-1.03-pl526_3, bioconda::perl-digest-md5-2.55-pl526_0, bioconda::perl-extutils-makemaker-7.36-pl526_1, bioconda::perl-html-tagset-3.20-pl526_3, bioconda::perl-io-html-1.001-pl526_2, bioconda::perl-io-zlib-1.10-pl526_2, conda-forge::perl-xml-parser-2.44_01-pl526ha1d75be_1002, bioconda::perl-json-4.02-pl526_0 path: 'lib/5.26.2/x86_64-linux-thread-multi/perllocal.pod'

ClobberError: This transaction has incompatible packages due to a shared path. packages: conda-forge::mmvec-1.0.2-py_0, conda-forge::songbird-1.0.1-py36_0 path: 'bin/qiime2.plugins' `

mortonjt commented 4 years ago

Ok, that looks like a conda error - maybe time to clean that up. See this SO answer on how to fix that.

ADGM commented 4 years ago

I've tried with conda clean --all and conda update --all, removing mmvec and installing fresh, and I'm still getting the error, unfortunately.

mortonjt commented 4 years ago

Hi @ADGM , ok -- then you probably installed it correctly.

It'll help to see what the command is. From a hunch, you maybe running the program with 0 iterations. Either your epochs is too low (i.e. 0) or your batch size is too high, or you are feeding in an empty dataset.

ADGM commented 4 years ago

Silly! It was because I was feeding a QZA feature table and a BIOM pathway table, so it was throwing an error in both qiime and python. I converted the biom into qza and it now works, thank you!

mijiandui commented 4 years ago

Ok, that looks like a conda error - maybe time to clean that up. See this SO answer on how to fix that.

Dear, mortonjt, i have the same error. But i installed following your advance and the files was also ok. I run with your sample data was fine. but an error with my data with 10 samples of two groups. Any idea where is wrong?

mortonjt commented 4 years ago

It's not clear why you are feeding this error, could you post more info (ie conda env export output?). But 10 samples is a bit too small to run MMvec.

On Sun, Mar 29, 2020, 9:45 AM mijiandui notifications@github.com wrote:

Ok, that looks like a conda error - maybe time to clean that up. See this SO https://stackoverflow.com/a/51600062/1167475 answer on how to fix that.

Dear, mortonjt, i have the same error. But i installed following your advance and the files was also ok. I run with your sample data was fine. but an error with my data with 10 samples of two groups. Any idea where is wrong?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocore/mmvec/issues/103#issuecomment-605638057, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXIOGAK2TN2YJ6TJDNTRJ5GF3ANCNFSM4JORYTUA .

mijiandui commented 4 years ago

It's not clear why you are feeding this error, could you post more info (ie conda env export output?). But 10 samples is a bit too small to run MMvec. …

The following are three conda envs export outputs. All had the same error:

Plugin error from mmvec:

local variable 'rU' referenced before assignment

Debug info has been saved to /tmp/qiime2-q2cli-err-lkk0cm2h.log.

So how many samples at least for MMvec analysis?

name: qiime2-2020.2 channels:

name: q2mmvec-env channels:

name: qiime2-2019.10 channels:

mortonjt commented 4 years ago

ok, based off of the previous error - these errors are thrown when the iterations are too small (i.e. the batch size is larger than the actual dataset).

Can you report the exact command that you are using, along with the total number of reads in your biom table? You could also confirm if this works by running with --p-batch-size 10.

mijiandui commented 4 years ago

ok, based off of the previous error - these errors are thrown when the iterations are too small (i.e. the batch size is larger than the actual dataset).

Can you report the exact command that you are using, along with the total number of reads in your biom table? You could also confirm if this works by running with --p-batch-size 10.

Dear, mortonjt, i run the following command and revised command with --p-batch-size 10. Same error. Total number of reads in my otu biom is about 221 reads and around 500 metabolites of meta. biom. Beacuse i think that error is because of too many zero in otu table and i filter the raw otus(with about 1900 reads). i will send the files i used. Thanks a lot for your help.

qiime mmvec paired-omics --i-microbes CeC-table-microbiome.qza --i-metabolites NEG.qza --o-conditionals ranks.qza --o-conditional-biplot biplot.qz --p-batch-size 10

mortonjt commented 4 years ago

Something seems off - the min number of reads you should expect is thousands per sample, not dozens...

If you have a sample with no reads, that will also cause this problem

On Sun, Mar 29, 2020, 10:31 AM mijiandui notifications@github.com wrote:

ok, based off of the previous error - these errors are thrown when the iterations are too small (i.e. the batch size is larger than the actual dataset).

Can you report the exact command that you are using, along with the total number of reads in your biom table? You could also confirm if this works by running with --p-batch-size 10.

Dear, mortonjt, i run the following command and revised command with --p-batch-size 10. Same error. Total number of reads in my otu biom is about 221 reads and around 500 metabolites of meta. biom. Beacuse i think that error is because of too many zero in otu table and i filter the raw otus(with about 1900 reads). i will send the files i used. Thanks a lot for your help.

qiime mmvec paired-omics --i-microbes CeC-table-microbiome.qza --i-metabolites NEG.qza --o-conditionals ranks.qza --o-conditional-biplot biplot.qz --p-batch-size 10

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocore/mmvec/issues/103#issuecomment-605645501, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXLTNRHT7GP5DGWAAETRJ5LT7ANCNFSM4JORYTUA .

mijiandui commented 4 years ago

Something seems off - the min number of reads you should expect is thousands per sample, not dozens... If you have a sample with no reads, that will also cause this problem …

Thanks. Even i used raw OTUs with 1,932 features, it is still error. Thanks. -- | --

mortonjt commented 4 years ago

feel free to post your tables - I'll be happy to see what stands out

mortonjt commented 4 years ago

alright, I'm looking at your tables and there are more problems

  1. You've got a ton of negative values in your metabolite table. MMvec assumes that your metabolites are in units of intensities, so it cannot handle negative values 
    >>> import qiime2
>>> import pandas as pd
>>> ms = qiime2.Artifact.load('CeC-NEG-metabolites.qza').view(pd.DataFrame)
>>> (ms<0).sum(axis=1).sum()
2740
  2. It is recommended to feed in your raw OTU table in instead of a table of proportions. MMvec uses the total number of reads to determine how long training is going to be. If you have only 10 samples, with each summing to 1, it thinks that you only have 10 reads across all of your samples.
mijiandui commented 4 years ago

CeC_OTU.biom.zip

alright, I'm looking at your tables and there are more problems

  1. You've got a ton of negative values in your metabolite table. MMvec assumes that your metabolites are in units of intensities, so it cannot handle negative values
>>> import qiime2
>>> import pandas as pd
>>> ms = qiime2.Artifact.load('CeC-NEG-metabolites.qza').view(pd.DataFrame)
>>> (ms<0).sum(axis=1).sum()
2740
  1. It is recommended to feed in your raw OTU table in instead of a table of proportions. MMvec uses the total number of reads to determine how long training is going to be. If you have only 10 samples, with each summing to 1, it thinks that you only have 10 reads across all of your samples.

Thanks. The negative values of metabolite table was the normalization one. I also use the raw counts of metabolite and otu table (not relative otu table) attached bellow. Same error. Please help to me check again. Thanks a lot. NEG.biom.zip CeC_OTU.biom.zip [Uploading CeC_OTU.biom.zip…]()

mortonjt commented 4 years ago

I was able run it just fine with the following command

mmvec paired-omics \
    --microbe-file CEC_OTU.biom \
    --metabolite-file NEG.biom \
    --min-feature-count 3 \
    --num-testing-examples 1 \
    --batch-size 10 \
    --summary-dir test

When you have so few samples, it is important to note the parameters --min-feature-count and --num-testing-examples, since those will further reduce the number of samples that you have. My guess is that the default --min-feature-count filtered out all of the OTUs (since the default is 10) and even if you fixed that --num-testing-examples would have allocated all of your samples to cross-validation.

So you should be able to get this working - but I'm not sure what results you hope to get out of this, I do not think this analysis will yield useful results with 10 samples.

mijiandui commented 4 years ago

I was able run it just fine with the following command

mmvec paired-omics \
    --microbe-file CEC_OTU.biom \
    --metabolite-file NEG.biom \
    --min-feature-count 3 \
    --num-testing-examples 1 \
    --batch-size 10 \
    --summary-dir test

When you have so few samples, it is important to note the parameters --min-feature-count and --num-testing-examples, since those will further reduce the number of samples that you have. My guess is that the default --min-feature-count filtered out all of the OTUs (since the default is 10) and even if you fixed that --num-testing-examples would have allocated all of your samples to cross-validation.

So you should be able to get this working - but I'm not sure what results you hope to get out of this, I do not think this analysis will yield useful results with 10 samples.

Dear, mortonjt, Thanks a lot. Following your advice, when i run the following command, that worked. I will check the result carefully and this information is also very important for my future study. Thanks again!!

qiime mmvec paired-omics --i-microbes CeC_OTU.qza --i-metabolites NEG.qza --o-onditionals ranks.qza --o-conditional-biplot biplot.qz --p-batch-size 10 --p-min-feature-count 3 --p-num-testing-examples 1