Open huajiachicat opened 1 year ago
huajiachicat
commented
1 year ago Is this issue caused by not enough memory? Since I run on HPC managed by Slurm, I allocated 1 node, 24 cores (CPUs), and 1T memory to this job. The VirSorter 2 is the most updated version so far.
jiarong
commented
1 year ago Hi, this issue has been seen before. It It happens when multiple hmmsearch processes run concurrently in certain OS, likely something to do hmmsearch memory management, but nothing to do with no enough allocated memory. The only solution I have now is to use the container (installation option 3).
huajiachicat
commented
1 year ago Hello:
Following your instruction, I downloaded the virsorter2.sif using singularity build virsorter2.sif docker://jiarong/virsorter:latestcommand. I did a test run and found another issue arises. This time the problem happened at Step 3.
Here is the error message (step3 only):
[2023-10-15 18:08 INFO] Step 2 - extract-feature finished.
[2023-10-15 18:08 ERROR] See error details in /users/POS0103/zzhao/local/virsorter2/testsif.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
[Sun Oct 15 18:08:51 2023]
Error in rule classify_by_group:
jobid: 56
output: iter-0/dsDNAphage/all.pdg.clf
conda-env: /users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79
shell:
Log=/users/POS0103/zzhao/local/virsorter2/testsif.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
python /usr/local/lib/python3.9/site-packages/virsorter/./scripts/classify.py iter-0/dsDNAphage/all.pdg.ftr /users/POS0103/zzhao/local/virsorter2/db/group/dsDNAphage/model dsDNAphage iter-0/dsDNAphage/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /usr/local/lib/python3.9/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
[2023-10-15 18:08 ERROR] See error details in /users/POS0103/zzhao/local/virsorter2/testsif.out/log/iter-0/step3-classify/all-score-ssDNA.log
[Sun Oct 15 18:08:51 2023]
Error in rule classify_by_group:
jobid: 57
output: iter-0/ssDNA/all.pdg.clf
conda-env: /users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79
shell:
Log=/users/POS0103/zzhao/local/virsorter2/testsif.out/log/iter-0/step3-classify/all-score-ssDNA.log
python /usr/local/lib/python3.9/site-packages/virsorter/./scripts/classify.py iter-0/ssDNA/all.pdg.ftr /users/POS0103/zzhao/local/virsorter2/db/group/ssDNA/model ssDNA iter-0/ssDNA/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /usr/local/lib/python3.9/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Exiting because a job execution failed. Look above for error message
*** An error occurred. Detailed errors may not be printed for certain rules. Refer to the log file of the failed command for troubleshootingI looked at the error message. Here is the content:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/virsorter/./scripts/classify.py", line 77, in <module>
main()
File "/usr/local/lib/python3.9/site-packages/virsorter/./scripts/classify.py", line 60, in main
model = joblib.load(model_f)
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 658, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 577, in _unpickle
obj = unpickler.load()
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/pickle.py", line 1212, in load
dispatch[key[0]](self)
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/pickle.py", line 1537, in load_stack_global
self.append(self.find_class(module, name))
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/pickle.py", line 1579, in find_class
__import__(module, level=0)
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/ensemble/__init__.py", line 7, in <module>
from ._forest import RandomForestClassifier
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/ensemble/_forest.py", line 56, in <module>
from ..tree import (DecisionTreeClassifier, DecisionTreeRegressor,
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/tree/__init__.py", line 6, in <module>
from ._classes import BaseDecisionTree
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/tree/_classes.py", line 40, in <module>
from ._criterion import Criterion
File "sklearn/tree/_splitter.pxd", line 34, in init sklearn.tree._criterion
File "sklearn/tree/_tree.pxd", line 37, in init sklearn.tree._splitter
File "sklearn/neighbors/_quad_tree.pxd", line 55, in init sklearn.tree._tree
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/neighbors/__init__.py", line 17, in <module>
from ._nca import NeighborhoodComponentsAnalysis
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/neighbors/_nca.py", line 22, in <module>
from ..decomposition import PCA
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/decomposition/__init__.py", line 17, in <module>
from .dict_learning import dict_learning
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/decomposition/dict_learning.py", line 4, in <module>
from . import _dict_learning
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/decomposition/_dict_learning.py", line 21, in <module>
from ..linear_model import Lasso, orthogonal_mp_gram, LassoLars, Lars
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/linear_model/__init__.py", line 12, in <module>
from ._least_angle import (Lars, LassoLars, lars_path, lars_path_gram, LarsCV,
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/sklearn/linear_model/_least_angle.py", line 30, in <module>
method='lar', copy_X=True, eps=np.finfo(np.float).eps,
File "/users/POS0103/zzhao/local/virsorter2/db/conda_envs/63e87a79/lib/python3.8/site-packages/numpy/__init__.py", line 305, in __getattr__
raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
It seems that it has a numpy version confiction.
Here is the problem: If I use option 2 develop version, then I will have a Hmmsearch issue. If I use the sif version, the numpy will have issue.
I run the sif version under VS2 python virtual environment. I tried to run it without activate VS2, same problem happened.
jiarong
commented
1 year ago This issue should gone after you remove ".virsorter" directory in your home directory.
Hello Jiarong: I installed and tested the developer's version (Option 2) Virsorter2 on HPC. It worked perfectly fine. However when I want to do viral search on my own dataset. It reports error at HMMsearch. Here is the part of the run log:
So I check the log file: /fs/scratch/POS0103/virsort2_analysis/apis_virsorter.out/iter-0/all.pdg.faa.splitdir/all.pdg.faa.ss.0.split.Viruses.splithmmtbl.log, here is the content: