Error in rule multiqc, missing package imp

cedarwarman commented 10 months ago

Hi, I'm getting an error in the multiqc step of the workflow when running it with the ecoli test dataset.

Here's a snippet of the error:

Error in rule multiqc:
    jobid: 1
    input: results/qc/fastqc/C00022105/C00022105_fastqc.zip, results/qc/fastqc/C00022106/C00022106_fastqc.zip, res
    output: results/qc/multiqc.html
    log: logs/qc/multiqc/multiqc.log (check log file(s) for error details)
    conda-env: /xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_

And the log file:

Traceback (most recent call last):
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/bin/multiqc", line 6, in <module>
    from multiqc.__main__ import multiqc
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/multiqc/__init__.py", line 16, in <module>
    from .multiqc import run
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/multiqc/multiqc.py", line 42, in <module>
    from .plots import table
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/multiqc/plots/table.py", line 9, in <module>
    from multiqc.utils import config, report, util_functions, mqc_colour
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/multiqc/utils/report.py", line 13, in <module>
    import lzstring
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/lzstring/__init__.py", line 11, in <module>
    from future import standard_library
  File "/xdisk/rpalaniv/cedar/kmers-gwas/ecoli_test_dir/.snakemake/conda/185bfc75953f1fe95746c2235e2af5f3_/lib/python3.12/site-packages/future/standard_library/__init__.py", line 65, in <module>
    import imp
ModuleNotFoundError: No module named 'imp'

I'm wondering if this is an HPC specific error, as I'm running it on my university cluster. Maybe the Conda environments aren't getting loaded?

cedarwarman commented 10 months ago

After doing some reading, I found this issue with MultiQC. I manually installed MultiQC version 1.17 inside the Conda environment and it successfully ran. Would a fix be as simple as updating the Dockerfile to use multiqc 1.17?

cedarwarman commented 10 months ago

Actually, I'm realizing the Dockerfile isn't actually used but rather the Conda environments are set up by snakemake-wrappers? Here is the commit in the multiqc snakemake-wrappers repo where they mention this bug. This was updated in snakemake-wrappers 2.8.0, so maybe the fix is to change this line of the qc rule to "v2.8.0/bio/multiqc"?

spaddys commented 7 months ago

I am also having this issue but the above solution did not solve the error for me, is there a different solution?

Thanks for any help!

Lemonadev3 commented 7 months ago

I am also having this issue but the above solution did not solve the error for me, is there a different solution?

Thanks for any help!

Have you solved it yet? I'm having error too .

spaddys commented 7 months ago

I am also having this issue but the above solution did not solve the error for me, is there a different solution? Thanks for any help!

Have you solved it yet? I'm having error too .

I haven't solved it yet, I'm still getting the same error.

cedarwarman commented 7 months ago

When I was troubleshooting this I found it really helpful to manually activate the Conda environments after they were created by the pipeline and then list the installed packages to see where it was going wrong. The paths to the environments are listed in the log files. Were you all able to confirm that the newer version of MultiQC was installed in the right Conda environment?

spaddys commented 7 months ago

When I was troubleshooting this I found it really helpful to manually activate the Conda environments after they were created by the pipeline and then list the installed packages to see where it was going wrong. The paths to the environments are listed in the log files. Were you all able to confirm that the newer version of MultiQC was installed in the right Conda environment?

I checked the MultiQC version but I'm wondering what version of Python worked for you?

cedarwarman commented 7 months ago

The Python version Conda used was 3.12 for that environment. Here's all the package versions:

[]✨  conda activate .snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_
(.snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_) []✨  conda list
# packages in environment at .snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
coloredlogs               15.0.1             pyhd8ed1ab_3    conda-forge
colormath                 3.0.0                      py_2    conda-forge
contourpy                 1.2.0           py312h8572e83_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
fonttools                 4.44.0          py312h98912ed_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
humanfriendly             10.0               pyhd8ed1ab_6    conda-forge
humanize                  4.8.0              pyhd8ed1ab_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
kiwisolver                1.4.5           py312h8572e83_1    conda-forge
lcms2                     2.15                 hb7c19ff_3    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           19_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           19_linux64_openblas    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           19_linux64_openblas    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.44.0               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lzstring                  1.0.4                   py_1001    conda-forge
markdown                  3.5.1              pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py312h98912ed_1    conda-forge
matplotlib-base           3.8.1           py312he5832f3_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
multiqc                   1.17               pyhdfd78af_1    bioconda
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
numpy                     1.26.0          py312heda63a1_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pillow                    10.1.0          py312hf3581a9_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.12.0          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pyyaml                    6.0.1           py312h98912ed_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rich                      13.6.0             pyhd8ed1ab_0    conda-forge
rich-click                1.7.1              pyhd8ed1ab_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
simplejson                3.19.2          py312h98912ed_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
spectra                   0.0.11                     py_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wheel                     0.41.3             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

spaddys commented 7 months ago

The Python version Conda used was 3.12 for that environment. Here's all the package versions:

[]✨  conda activate .snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_
(.snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_) []✨  conda list
# packages in environment at .snakemake/conda/4f2426c013f70c176b858dce0c5a5cfe_:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
coloredlogs               15.0.1             pyhd8ed1ab_3    conda-forge
colormath                 3.0.0                      py_2    conda-forge
contourpy                 1.2.0           py312h8572e83_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
fonttools                 4.44.0          py312h98912ed_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
humanfriendly             10.0               pyhd8ed1ab_6    conda-forge
humanize                  4.8.0              pyhd8ed1ab_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
kiwisolver                1.4.5           py312h8572e83_1    conda-forge
lcms2                     2.15                 hb7c19ff_3    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           19_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           19_linux64_openblas    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           19_linux64_openblas    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.44.0               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lzstring                  1.0.4                   py_1001    conda-forge
markdown                  3.5.1              pyhd8ed1ab_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py312h98912ed_1    conda-forge
matplotlib-base           3.8.1           py312he5832f3_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
multiqc                   1.17               pyhdfd78af_1    bioconda
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
numpy                     1.26.0          py312heda63a1_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pillow                    10.1.0          py312hf3581a9_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.12.0          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pyyaml                    6.0.1           py312h98912ed_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rich                      13.6.0             pyhd8ed1ab_0    conda-forge
rich-click                1.7.1              pyhd8ed1ab_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
simplejson                3.19.2          py312h98912ed_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
spectra                   0.0.11                     py_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wheel                     0.41.3             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Thank you! Your fix solved that issue for me.

cedarwarman commented 7 months ago

Hooray!!

akcorut / kGWASflow

Error in rule multiqc, missing package imp #26