search

jiarong / VirSorter2

customizable pipeline to identify viral sequences from (meta)genomic data
GNU General Public License v2.0
219 stars 29 forks source link

Error in rule classify_by_group #156

Open dangchenyuan opened 1 year ago

dangchenyuan commented 1 year ago

Hi jiarong, I have met this error "Error in rule classify_by_group" when I run the vs. After scanning the pre-answers, I tried all three options to install the vs2. But they all reported the same error, even including the Apptainer method, and the command was "virsorter2.sif run -w test.out -i test.fa --min-length 5000 -j 40 all".

The log file is as follows: image

jiarong commented 1 year ago

Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be NOT in the apptainer version.

dangchenyuan commented 1 year ago

Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be in the apptainer version.

Yes, '/home/dcy/' is my user's path. My apptainer version is 1.1.6.

TKsh6 commented 1 year ago

I have met the same error.

[2023-03-29 23:18 ERROR] See error details in /beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-ssDNA.log
[Wed Mar 29 23:18:01 2023]
Error in rule classify_by_group:
    jobid: 57
    output: iter-0/ssDNA/all.pdg.clf
    shell:

        Log=/beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-ssDNA.log
        python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py iter-0/ssDNA/all.pdg.ftr /beegfs/home/syl/database_db/vir/group/ssDNA/model ssDNA iter-0/ssDNA/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[2023-03-29 23:18 ERROR] See error details in /beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
[Wed Mar 29 23:18:01 2023]
Error in rule classify_by_group:
    jobid: 56
    output: iter-0/dsDNAphage/all.pdg.clf
    shell:

        Log=/beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
        python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py iter-0/dsDNAphage/all.pdg.ftr /beegfs/home/syl/database_db/vir/group/dsDNAphage/model dsDNAphage iter-0/dsDNAphage/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Exiting because a job execution failed. Look above for error message

*** An error occurred. Detailed errors may not be printed for certain rules. Refer to the log file of the failed command for troubleshooting
Issues can be raised at: https://github.com/jiarong/VirSorter2/issues
Or send an email to virsorter2 near gmail.com if you do not use GitHub

all-score-ssDNA.log as follows

/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator Pipeline from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names
  f"X has feature names, but {self.__class__.__name__} was fitted without"
Traceback (most recent call last):
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 77, in <module>
    main()
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 68, in main
    pred_prob = model.predict_proba(X)
  File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/utils/metaestimators.py", line 113, in <lambda>
    out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs)  # noqa
  File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/pipeline.py", line 535, in predict_proba
    Xt = transform.transform(Xt)
  File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/preprocessing/_data.py", line 506, in transform
    if self.clip:
AttributeError: 'MinMaxScaler' object has no attribute 'clip'
jiarong commented 1 year ago

Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be in the apptainer version.

Yes, '/home/dcy/' is my user's path. My apptainer version is 1.1.6.

@dangchenyuan Sorry I had typo in my last reply. I meant to say the error output does not look like from apptainer version, which should NOT have your user's path. Please double check your run with apptainer version.

jiarong commented 1 year ago

@TKsh6 what's the commend line? and other parts of the output message?

TKsh6 commented 1 year ago

@jiarong I have changed the environment path of python, but I still got the same problem. my commend line: virsorter run -w test.out -i test.fa --min-length 1500 -j 4 all -d /beegfs/home/syl/database_db/vir --use-conda-off the log:

/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator Pipeline from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:432: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names
  warnings.warn(
Traceback (most recent call last):
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 77, in <module>
    main()
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 68, in main
    pred_prob = model.predict_proba(X)
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/pipeline.py", line 546, in predict_proba
    Xt = transform.transform(Xt)
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 140, in wrapped
    data_to_wrap = f(self, X, *args, **kwargs)
  File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/preprocessing/_data.py", line 518, in transform
    if self.clip:
AttributeError: 'MinMaxScaler' object has no attribute 'clip'
TKsh6 commented 1 year ago

@jiarong and this are the version of the packages,

(vs2) [syl@login test_forvir]$ conda list
# packages in environment at /beegfs/home/syl/anaconda3/envs/vs2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
amply                     0.1.5              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
apptainer                 1.1.7                h58603c9_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
binaryornot               0.4.4                      py_1    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bz2file                   0.98                       py_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
chardet                   5.1.0           py310hff52083_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
cni                       1.0.1                ha975731_1    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
cni-plugins               1.0.1                ha8f183a_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
coin-or-cbc               2.10.8               h3786ebc_0    conda-forge
coin-or-cgl               0.60.6               h6f57e76_2    conda-forge
coin-or-clp               1.17.7               hc56784d_2    conda-forge
coin-or-osi               0.108.7              h2720bb7_2    conda-forge
coin-or-utils             2.11.6               h202d8b1_2    conda-forge
coincbc                   2.10.8            0_metapackage    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.3.0          py310hff52083_0    conda-forge
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.7.0              pyhd8ed1ab_1    conda-forge
configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
cookiecutter              2.1.1              pyh6c4a22f_0    conda-forge
cryptography              40.0.1          py310h34c0648_0    conda-forge
curl                      7.88.1               hdc1c0ab_1    conda-forge
datrie                    0.8.2           py310h5764c6d_6    conda-forge
docutils                  0.19            py310hff52083_1    conda-forge
expat                     2.5.0                h27087fc_0    conda-forge
fmt                       9.1.0                h924138e_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
git                       2.40.0          pl5321h3ca6407_1    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.1.0              pyha770c72_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jinja2-time               0.2.0              pyhd8ed1ab_3    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
jq                        1.5                           0    bioconda
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.0           py310hff52083_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.6.2                h3d51595_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcurl                   7.88.1               hdc1c0ab_1    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
liblapacke                3.9.0           16_linux64_openblas    conda-forge
libmamba                  1.4.1                hcea66bb_0    conda-forge
libmambapy                1.4.1           py310h1428755_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libseccomp                2.4.4                h166bdaf_3    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
libsolv                   0.7.23               h3eb15da_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libssh2                   1.10.0               hf14f497_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.10.3               hfdac1af_6    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.4.1           py310h51d5547_0    conda-forge
markupsafe                2.1.2           py310h1fa729e_0    conda-forge
nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.24.2          py310h8deb116_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
openssl                   3.1.0                h0b41bf4_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pandas                    1.5.3           py310h9b08913_1    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
perl                      5.32.1          2_h7f98852_perl5    conda-forge
pip                       23.0.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.2.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
pooch                     1.7.0              pyha770c72_3    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
psutil                    5.9.4           py310h5764c6d_0    conda-forge
pulp                      2.7.0           py310hff52083_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5764c6d_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.10         he550d4f_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.3             pyhd8ed1ab_0    conda-forge
python-slugify            8.0.1              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
ratelimiter               1.2.0           pyhd8ed1ab_1003    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.28.2             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.16.12         py310h5764c6d_3    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
scikit-learn              1.2.2           py310h41b6a48_1    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
scipy                     1.10.1          py310h8deb116_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
screed                    1.0.4                      py_0    bioconda
setuptools                67.6.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake-minimal         5.26.0                     py_0    bioconda
squashfs-tools            4.4                  hd0129a2_3    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
text-unidecode            1.3                        py_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    https://mirrors.tuna.tsingh                              ua.edu.cn/anaconda/cloud/conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
toposort                  1.10               pyhd8ed1ab_0    conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
typing_extensions         4.5.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unidecode                 1.3.6              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
virsorter                 2.2.4              pyhdfd78af_0    bioconda
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0          py310h1fa729e_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.19.0          py310hdeb6495_1    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge
jiarong commented 1 year ago

@dangchenyuan see #157 if you are in the same situation.

TKsh6 commented 1 year ago

@jiarong So I can't use it through conda, only through docker?

jiarong commented 1 year ago

@TKsh6 If you reinstall your OS, conda version should work. If you do not want to reinstall OS, it's too complicated for me to find out the specific problem that is different for different users. That's why I recommend the apptainer/docker version.

TKsh6 commented 1 year ago

it's not convenient to perform batch work through docker version, every time I need to ssh the specific node for running.

jiarong commented 1 year ago

Apptainer is not available on all nodes? It's typically installed to be available server wide. You can ask you server admin to help.

TKsh6 commented 1 year ago

Yes, you are right. I have solved this both in conda. Only need to install scikit-learn=0.22.1. Thanks a lot.

dangchenyuan commented 1 year ago

Hi jiarong, I have solved it now. Because of the conflict with different sklearn versions.