search

InternLM / xtuner

An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)
https://xtuner.readthedocs.io/zh-cn/latest/
Apache License 2.0
3.76k stars 302 forks source link

AttributeError: module 'cv2.dnn' has no attribute 'DictValue' #115

Closed qingqiuhe closed 1 year ago

qingqiuhe commented 1 year ago 
root@localhost:/workspace/xtuner# pip install -e '.[all]'
root@localhost:/workspace/xtuner# xtuner
[2023-09-11 15:13:59,638] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/usr/local/bin/xtuner", line 33, in <module>
    sys.exit(load_entry_point('xtuner', 'console_scripts', 'xtuner')())
  File "/usr/local/bin/xtuner", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspace/xtuner/xtuner/__init__.py", line 4, in <module>
    from .entry_point import cli
  File "/workspace/xtuner/xtuner/entry_point.py", line 11, in <module>
    from xtuner.tools import (chat, check_custom_dataset, copy_cfg, list_cfg,
  File "/workspace/xtuner/xtuner/tools/test.py", line 9, in <module>
    from mmengine.runner import Runner
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/__init__.py", line 2, in <module>
    from ._flexible_runner import FlexibleRunner
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/_flexible_runner.py", line 14, in <module>
    from mmengine._strategy import BaseStrategy
  File "/usr/local/lib/python3.10/dist-packages/mmengine/_strategy/__init__.py", line 3, in <module>
    from mmengine.utils.dl_utils import TORCH_VERSION
  File "/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/__init__.py", line 3, in <module>
    from .collect_env import collect_env
  File "/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/collect_env.py", line 8, in <module>
    import cv2
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 175, in bootstrap
    if __load_extra_py_code_for_module("cv2", submodule, DEBUG):
  File "/usr/local/lib/python3.10/dist-packages/cv2/__init__.py", line 28, in __load_extra_py_code_for_module
    py_module = importlib.import_module(module_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.10/dist-packages/cv2/typing/__init__.py", line 169, in <module>
    LayerId = cv2.dnn.DictValue
AttributeError: module 'cv2.dnn' has no attribute 'DictValue'
LZHgrla commented 1 year ago

Could you please paste the environment information? You can obtain it by pip list

qingqiuhe commented 1 year ago

Could you please paste the environment information? You can obtain it by pip list

Package                       Version              Editable project location
----------------------------- -------------------- -------------------------
absl-py                       1.4.0
accelerate                    0.22.0
addict                        2.4.0
aiofiles                      23.2.1
aiohttp                       3.8.4
aiosignal                     1.3.1
altair                        5.1.1
anyio                         3.7.1
apex                          0.1
appdirs                       1.4.4
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
asttokens                     2.2.1
astunparse                    1.6.3
async-timeout                 4.0.2
attrs                         23.1.0
audioread                     3.0.0
backcall                      0.2.0
beautifulsoup4                4.12.2
bitsandbytes                  0.41.0
black                         23.3.0
bleach                        6.0.0
blis                          0.7.10
cachetools                    5.3.1
catalogue                     2.0.9
certifi                       2023.7.22
cffi                          1.15.1
charset-normalizer            3.2.0
click                         8.1.5
cloudpickle                   2.2.1
cmake                         3.27.1
comm                          0.1.4
confection                    0.1.1
contourpy                     1.1.0
cpm-kernels                   1.0.11
cubinlinker                   0.3.0+2.g7c3675e
cuda-python                   12.1.0rc5+1.g994d8d0
cudf                          23.6.0
cugraph                       23.6.0
cugraph-dgl                   23.6.0
cugraph-service-client        23.6.0
cugraph-service-server        23.6.0
cuml                          23.6.0
cupy-cuda12x                  12.1.0
cycler                        0.11.0
cymem                         2.0.7
Cython                        3.0.0
dask                          2023.3.2
dask-cuda                     23.6.0
dask-cudf                     23.6.0
datasets                      2.14.4
dbus-python                   1.2.18
debugpy                       1.6.7
decorator                     5.1.1
deepspeed                     0.10.2
defusedxml                    0.7.1
dill                          0.3.7
distributed                   2023.3.2.1
distro                        1.7.0
dm-tree                       0.1.8
einops                        0.6.1
exceptiongroup                1.1.2
execnet                       2.0.2
executing                     1.2.0
expecttest                    0.1.3
fastapi                       0.95.1
fastjsonschema                2.18.0
fastrlock                     0.8.1
ffmpy                         0.3.1
filelock                      3.12.2
fire                          0.5.0
flash-attn                    2.0.4
fonttools                     4.42.0
frozenlist                    1.4.0
fsspec                        2023.6.0
gast                          0.5.4
google-auth                   2.22.0
google-auth-oauthlib          0.4.6
gradio                        3.42.0
gradio_client                 0.5.0
graphsurgeon                  0.4.6
grpcio                        1.56.2
h11                           0.14.0
hjson                         3.1.0
httpcore                      0.17.3
httpx                         0.24.1
huggingface-hub               0.16.4
hypothesis                    5.35.1
idna                          3.4
importlib-metadata            6.8.0
importlib-resources           6.0.1
iniconfig                     2.0.0
intel-openmp                  2021.4.0
ipykernel                     6.25.0
ipython                       8.14.0
ipython-genutils              0.2.0
jedi                          0.19.0
jieba                         0.42.1
Jinja2                        3.1.2
joblib                        1.3.1
json5                         0.9.14
jsonschema                    4.18.6
jsonschema-specifications     2023.7.1
jupyter_client                8.3.0
jupyter_core                  5.3.1
jupyter-tensorboard           0.2.0
jupyterlab                    2.3.2
jupyterlab-pygments           0.2.2
jupyterlab-server             1.2.0
jupytext                      1.15.0
kiwisolver                    1.4.4
langcodes                     3.3.0
librosa                       0.9.2
llvmlite                      0.40.1
locket                        1.0.0
Markdown                      3.4.4
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
matplotlib                    3.7.2
matplotlib-inline             0.1.6
mdit-py-plugins               0.4.0
mdurl                         0.1.2
mistune                       3.0.1
mkl                           2021.1.1
mkl-devel                     2021.1.1
mkl-include                   2021.1.1
mmengine                      0.8.4
mock                          5.1.0
mpi4py-mpich                  3.1.2
mpmath                        1.3.0
msgpack                       1.0.5
multidict                     6.0.4
multiprocess                  0.70.15
murmurhash                    1.0.9
mypy-extensions               1.0.0
nbclient                      0.8.0
nbconvert                     7.7.3
nbformat                      5.9.2
nest-asyncio                  1.5.7
networkx                      2.6.3
ninja                         1.11.1
nltk                          3.8.1
notebook                      6.4.10
numba                         0.57.1+1.gc785c8f1f
numpy                         1.23.5
nvidia-dali-cuda120           1.28.0
nvidia-pyindex                1.0.9
nvtx                          0.2.5
oauthlib                      3.2.2
onnx                          1.14.0
opencv                        4.7.0
opencv-python                 4.8.0.76
orjson                        3.9.5
packaging                     23.1
pandas                        1.5.3
pandocfilters                 1.5.0
parso                         0.8.3
partd                         1.4.0
pathspec                      0.11.2
pathy                         0.10.2
peft                          0.5.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.2.0
pip                           23.2.1
platformdirs                  3.10.0
pluggy                        1.2.0
ply                           3.11
polygraphy                    0.47.1
pooch                         1.7.0
preshed                       3.0.8
prettytable                   3.8.0
prometheus-client             0.17.1
prompt-toolkit                3.0.39
protobuf                      4.21.12
psutil                        5.9.4
ptxcompiler                   0.8.1+1.g4a94326
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyarrow                       11.0.0
pyasn1                        0.5.0
pyasn1-modules                0.3.0
pybind11                      2.11.1
pycocotools                   2.0+nv0.7.3
pycparser                     2.21
pydantic                      1.10.11
pydub                         0.25.1
Pygments                      2.16.1
PyGObject                     3.42.1
pylibcugraph                  23.6.0
pylibcugraphops               23.6.0
pylibraft                     23.6.0
pynvml                        11.4.1
pyparsing                     3.0.9
pytest                        7.4.0
pytest-flakefinder            1.1.0
pytest-rerunfailures          12.0
pytest-shard                  0.1.2
pytest-xdist                  3.3.1
python-dateutil               2.8.2
python-dotenv                 1.0.0
python-hostlist               1.23.0
python-multipart              0.0.6
pytorch-quantization          2.1.2
pytz                          2023.3
PyYAML                        6.0.1
pyzmq                         25.1.0
raft-dask                     23.6.0
referencing                   0.30.2
regex                         2023.6.3
requests                      2.31.0
requests-oauthlib             1.3.1
resampy                       0.4.2
rich                          13.5.2
rmm                           23.6.0
rouge-chinese                 1.0.3
rpds-py                       0.9.2
rsa                           4.9
safetensors                   0.3.3
scikit-learn                  1.2.0
scipy                         1.11.1
semantic-version              2.10.0
Send2Trash                    1.8.2
sentencepiece                 0.1.99
setuptools                    68.0.0
six                           1.16.0
smart-open                    6.3.0
sniffio                       1.3.0
sortedcontainers              2.4.0
soundfile                     0.12.1
soupsieve                     2.4.1
spacy                         3.6.0
spacy-legacy                  3.0.12
spacy-loggers                 1.0.4
sphinx-glpi-theme             0.3
srsly                         2.4.7
sse-starlette                 1.6.5
ssh-import-id                 5.11
stack-data                    0.6.2
starlette                     0.26.1
sympy                         1.12
tabulate                      0.9.0
tbb                           2021.10.0
tblib                         2.0.0
tensorboard                   2.9.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorrt                      8.6.1
termcolor                     2.3.0
terminado                     0.17.1
thinc                         8.1.10
threadpoolctl                 3.2.0
thriftpy2                     0.4.16
tiktoken                      0.4.0
tinycss2                      1.2.1
tokenize-rt                   5.2.0
tokenizers                    0.13.3
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.0
torch                         2.1.0a0+29c30b1
torch-tensorrt                2.0.0.dev0
torchdata                     0.7.0a0
torchtext                     0.16.0a0
torchvision                   0.16.0a0
tornado                       6.3.2
tqdm                          4.65.0
traitlets                     5.9.0
transformer-engine            0.11.0+3f01b4f
transformers                  4.32.1
transformers-stream-generator 0.0.4
treelite                      3.2.0
treelite-runtime              3.2.0
triton                        2.1.0+440fd1b
trl                           0.7.1
typer                         0.9.0
types-dataclasses             0.6.6
typing_extensions             4.5.0
ucx-py                        0.32.0
uff                           0.6.9
urllib3                       1.26.16
uvicorn                       0.23.2
wasabi                        1.1.2
wcwidth                       0.2.6
webencodings                  0.5.1
websockets                    11.0.3
Werkzeug                      2.3.6
wheel                         0.41.1
xdoctest                      1.0.2
xgboost                       1.7.5
xtuner                        0.1.3                /workspace/xtuner
xxhash                        3.3.0
yapf                          0.40.1
yarl                          1.9.2
zict                          3.0.0
zipp                          3.16.2
LZHgrla commented 1 year ago

opencv 4.7.0 opencv-python 4.8.0.76

It seems that you have two opencv lib, try to uninstall opencv by pip uninstall opencv?

qingqiuhe commented 1 year ago

It works! Thx

qingqiuhe commented 1 year ago

anthor error when i start training

opencv 4.7.0 opencv-python 4.8.0.76

It seems that you have two opencv lib, try to uninstall opencv by pip uninstall opencv?

anthor error when i start training

# CUDA_VISIBLE_DEVICES=1 xtuner train llama2_7b_chat_qlora_sql_e3_copy.py
[2023-09-12 17:07:15,673] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/workspace/xtuner/xtuner/tools/train.py", line 246, in <module>
    main()
  File "/workspace/xtuner/xtuner/tools/train.py", line 235, in main
    runner = Runner.from_cfg(cfg)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/runner.py", line 445, in from_cfg
    runner = cls(
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/runner.py", line 355, in __init__
    self.setup_env(env_cfg)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/runner.py", line 653, in setup_env
    set_multi_processing(**mp_cfg, distributed=self.distributed)
  File "/usr/local/lib/python3.10/dist-packages/mmengine/utils/dl_utils/setup_env.py", line 38, in set_multi_processing
    cv2.setNumThreads(opencv_num_threads)
AttributeError: module 'cv2' has no attribute 'setNumThreads'
crownwang13 commented 2 months ago

uninstall opencv-python and then install opencv-python with the same version as opencv I have tried and it works eg. before , my version of opencv is 4.7.0,and opencv-python is 4.10.0.84 and in order to solve this error, then you can pip uninstall opencv-python pip install opencv-python==4.7.0.72