Current version of moshpit incompatible with conda version of q2-types and q2-types-genomics.

Let's assume you want to implement some new feature in moshpit. You would fork this repo and then clone the fork to your local machine where you would work on the new features. But before you start coding you need to set up the virtual environment. So you follow the instructions in the wiki and run the following (notice how moshpit is left out of the mamba create command since you will install your local copy and not the version that is available through mamba).

mamba create -yn test_env \
    -c conda-forge -c bioconda -c https://packages.qiime2.org/qiime2/2023.5/tested -c defaults \
    q2cli q2-assembly q2-checkm

conda run -n test_env \
    pip install --no-deps --force-reinstall git+https://github.com/misialq/quast.git@issue-230

conda activate test_env

cd <path_to_local_moshpit_repo>/q2-moshpit

pip install -e .

So far so good, but then running anything (e.g. qiime dev refresh-cache) will return this error:

ImportError: cannot import name 'BrackenDBDirectoryFormat' from 'q2_types_genomics.kraken2'

This error comes from q2_types_genomics/kraken2/__init__.py (and perhaps the other files in q2_types_genomics/kraken2). The version that gets installed is missing some classes that are used by moshpit.

Installed version:

 from ._format import (
     Kraken2ReportFormat, Kraken2ReportDirectoryFormat,
     Kraken2OutputFormat, Kraken2OutputDirectoryFormat,
     Kraken2DBFormat, Kraken2DBDirectoryFormat
 )
 from ._type import Kraken2Reports, Kraken2Outputs, Kraken2DB

 __all__ = [
     'Kraken2ReportFormat', 'Kraken2ReportDirectoryFormat', 'Kraken2Reports',
     'Kraken2OutputFormat', 'Kraken2OutputDirectoryFormat', 'Kraken2Outputs',
     'Kraken2DBFormat', 'Kraken2DBDirectoryFormat', 'Kraken2DB'
 ]

However, these classes are available in the current version of q2-types-genomics.

Current GitHub version:

from ._format import (
    Kraken2ReportFormat, Kraken2ReportDirectoryFormat,
    Kraken2OutputFormat, Kraken2OutputDirectoryFormat,
    Kraken2DBFormat, Kraken2DBDirectoryFormat,
    BrackenDBFormat, BrackenDBDirectoryFormat
)
from ._type import Kraken2Reports, Kraken2Outputs, Kraken2DB

__all__ = [
    'Kraken2ReportFormat', 'Kraken2ReportDirectoryFormat', 'Kraken2Reports',
    'Kraken2OutputFormat', 'Kraken2OutputDirectoryFormat', 'Kraken2Outputs',
    'Kraken2DBFormat', 'Kraken2DBDirectoryFormat', 'Kraken2DB',
    'BrackenDBFormat', 'BrackenDBDirectoryFormat'
]

link to lines of code in q2-types-genomics

So we can fix it by running the following.

pip uninstall q2_types_genomics
git clone git@github.com:bokulich-lab/q2-types-genomics.git
cd <path_to_local_q2-types-genomics_repo>/q2-types-genomics
pip install .

Ok problem solved! Sadly not, apparently q2-types-genomics is not the only incompatible package. If we run qiime dev refresh-cache we will get the following error.

TypeError: BLAST6 is not a variant of SampleData.field['type']

The error comes from the q2-types package. It boils down to the q2_types/feature_data/_type.py lines 51-53.

Old version (installed with conda/mamba):

BLAST6 = SemanticType('BLAST6',
                      variant_of=[FeatureData.field['type']])

New version (available in GitHub):

BLAST6 = SemanticType('BLAST6',
                      variant_of=[FeatureData.field['type'],
                                  SampleData.field['type']])

link to lines of code in q2-types

Again we uninstall the guilty package and clone the new version and install it locally.

pip uninstall q2-types
git clone https://github.com/qiime2/q2-types.git
cd <path_to_local_q2-types>/q2-types
pip install .

Woohoo! We solved it! Almost. Running qiime dev refresh-cache will return import errors, essentially complaining that some libraries are missing from the environment. Just

pip install xmltodict tqdm

(the libraries that are missing) and you are good to go.

bokulich-lab / q2-moshpit

Current version of moshpit incompatible with conda version of q2-types and q2-types-genomics. #58