biocore / songbird

Vanilla regression methods for microbiome differential abundance analysis
BSD 3-Clause "New" or "Revised" License
54 stars 25 forks source link

Plugin error from songbird: 'Testing' #150

Closed bck243 closed 3 years ago

bck243 commented 3 years ago

Hello,

I recently closed an issue about getting the warning warnings.warn('Cannot determine yajl version, assuming <1.0.12') when running the songbird tutorial with qiime version 2020.6. I was able to get the tutorial to work, but the warning always popped up.

Now, running songbird on my own data, I'm still getting the warning and I think it may be the reason songbird won't run.

Here's what I did:

qiime songbird multinomial \
  --i-table ./data/P18_16S_all_runs_raw_no_contam_or_control2.qza \
  --m-metadata-file ./data/iTAG_metadata_16S_all_for_q2_8_2020.tsv \
  --p-formula "CTDPRS" \
  --p-epochs 10000 \
  --p-differential-prior 0.5 \
  --p-training-column Testing \
  --p-summary-interval 1 \
  --o-differentials CTDPRS_differentials.qza \
  --o-regression-stats CTDPRS_regression-stats.qza \
  --o-regression-biplot CTDPRS_regression-biplot.qza

Here's the error I get:

/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/ijson/backends/__init__.py:22: UserWarning: Cannot determine yajl version, assuming <1.0.12
  warnings.warn('Cannot determine yajl version, assuming <1.0.12')
Plugin error from songbird:

  'Testing'

Debug info has been saved to /usr/local/scratch/path/tmp/qiime2-q2cli-err-c8tu95jk.log

And here's the error log:

Traceback (most recent call last):
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Testing'

During handling of the above exception, another exception occurred:
(qiime2-2020.6) Fri Jan 22 14:46:48 bkolody@lserver1 /usr/local/path/P18/process_ribotag/songbird/P18_16S
> cat /usr/local/scratch/METAGENOMICS/bkolody/tmp/qiime2-q2cli-err-c8tu95jk.log
Traceback (most recent call last):
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Testing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-121>", line 2, in multinomial
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/songbird/q2/_method.py", line 53, in multinomial
    seed=random_seed,
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/songbird/util.py", line 199, in split_training
    train_idx = metadata.loc[design.index, training_column] == "Train"
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1418, in __getitem__
    return self._getitem_tuple(key)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 805, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 929, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1850, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 160, in _get_label
    return self.obj._xs(label, axis=axis)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/generic.py", line 3729, in xs
    return self[key]
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Testing'

Any help would be appreciated!

mortonjt commented 3 years ago

This error is basically saying that your sample metadata file doesn't have a Testing column. You can confirm this if you remove this option.

So likely you either need to add that column, or you don't use it (which it will randomly split your samples into training and testing).

On Fri, Jan 22, 2021 at 12:57 PM bck243 notifications@github.com wrote:

Hello,

I recently closed an issue https://github.com/biocore/songbird/issues/149 about getting the warning warnings.warn('Cannot determine yajl version, assuming <1.0.12') when running the songbird tutorial with qiime version 2020.6. I was able to get the tutorial to work, but the warning always popped up.

Now, running songbird on my own data, I'm still getting the warning and I think it may be the reason songbird won't run.

Here's what I did:

qiime songbird multinomial \ --i-table ./data/P18_16S_all_runs_raw_no_contam_or_control2.qza \ --m-metadata-file ./data/iTAG_metadata_16S_all_for_q2_8_2020.tsv \ --p-formula "CTDPRS" \ --p-epochs 10000 \ --p-differential-prior 0.5 \ --p-training-column Testing \ --p-summary-interval 1 \ --o-differentials CTDPRS_differentials.qza \ --o-regression-stats CTDPRS_regression-stats.qza \ --o-regression-biplot CTDPRS_regression-biplot.qza

Here's the error I get:

/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/ijson/backends/init.py:22: UserWarning: Cannot determine yajl version, assuming <1.0.12 warnings.warn('Cannot determine yajl version, assuming <1.0.12') Plugin error from songbird:

'Testing'

Debug info has been saved to /usr/local/scratch/path/tmp/qiime2-q2cli-err-c8tu95jk.log

And here's the error log:

Traceback (most recent call last): File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Testing'

During handling of the above exception, another exception occurred: (qiime2-2020.6) Fri Jan 22 14:46:48 bkolody@lserver1 /usr/local/path/P18/process_ribotag/songbird/P18_16S

cat /usr/local/scratch/METAGENOMICS/bkolody/tmp/qiime2-q2cli-err-c8tu95jk.log Traceback (most recent call last): File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Testing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call results = action(arguments) File "", line 2, in multinomial File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable output_types, provenance) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callableexecutor output_views = self._callable(view_args) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/songbird/q2/_method.py", line 53, in multinomial seed=random_seed, File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/songbird/util.py", line 199, in split_training train_idx = metadata.loc[design.index, training_column] == "Train" File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1418, in getitem return self._getitem_tuple(key) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 805, in _getitem_tuple return self._getitem_lowerdim(tup) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 929, in _getitem_lowerdim section = self._getitem_axis(key, axis=i) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 1850, in _getitem_axis return self._get_label(key, axis=axis) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexing.py", line 160, in _get_label return self.obj._xs(label, axis=axis) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/generic.py", line 3729, in xs return self[key] File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/frame.py", line 2995, in getitem indexer = self.columns.get_loc(key) File "/usr/local/path/installations/miniconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Testing'

Any help would be appreciated!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/biocore/songbird/issues/150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXPSABAEFMT67CZJ2A3S3HKBBANCNFSM4WO67KVA .

bck243 commented 3 years ago

Thanks @mortonjt! Got it. It's working now but taking awhile to run. Is it possible to multithread it? If so, how do I specify how many threads I've allocated on the server?

mortonjt commented 3 years ago

It should by default use all of the cores available.

So if you are using a slurm cluster, you'll want to specify the number of cores your job is allowed to access.

On Mon, Jan 25, 2021 at 10:49 AM bck243 notifications@github.com wrote:

Thanks @mortonjt https://github.com/mortonjt! Got it. It's working now but taking awhile to run. Is it possible to multithread it? If so, how do I specify how many threads I've allocated on the server?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/biocore/songbird/issues/150#issuecomment-766996071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXJV5XQTEKIFT56O3PDS3WVKHANCNFSM4WO67KVA .

bck243 commented 3 years ago

Great, thank you!