search

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)
https://arxiv.org/abs/2403.13372
Apache License 2.0
33.25k stars 4.09k forks source link

最新的代码在 V100 机器，合并 bf16 的 base model 的 LoRA 权重时会报 addmm_impl_cpu_" not implemented for 'Half #3434

Closed lvsijian8 closed 6 months ago

lvsijian8 commented 6 months ago

Reminder

Reproduction

CUDA_VISIBLE_DEVICES='0' python src/export_model.py

报错： 04/25/2024 14:57:42 - INFO - llmtuner.model.adapter - Fine-tuning method: LoRA Traceback (most recent call last): File "/home/work/chatbot-ds-common-3/sijian.lv/projects/LLM/LLaMA-Factory/src/export_model.py", line 9, in main() File "/home/work/chatbot-ds-common-3/sijian.lv/projects/LLM/LLaMA-Factory/src/export_model.py", line 5, in main export_model() File "/home/work/chatbot-ds-common-3/sijian.lv/projects/LLM/LLaMA-Factory/src/llmtuner/train/tuner.py", line 57, in export_model model = load_model(tokenizer, model_args, finetuning_args) # must after fixing tokenizer to resize vocab File "/home/work/chatbot-ds-common-3/sijian.lv/projects/LLM/LLaMA-Factory/src/llmtuner/model/loader.py", line 106, in load_model model = init_adapter(model, model_args, finetuning_args, is_trainable) File "/home/work/chatbot-ds-common-3/sijian.lv/projects/LLM/LLaMA-Factory/src/llmtuner/model/adapter.py", line 113, in init_adapter model: "LoraModel" = PeftModel.from_pretrained( File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 355, in from_pretrained model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[config.task_type](model, config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 1094, in init super().init(model, peft_config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/peft_model.py", line 129, in init self.base_model = cls(model, {adapter_name: peft_config}, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 136, in init super().init(model, config, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 148, in init self.inject_adapter(self.model, adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/tuners_utils.py", line 325, in inject_adapter self._create_and_replace(peft_config, adapter_name, target, target_name, parent, current_key=key) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 220, in _create_and_replace new_module = self._create_new_module(lora_config, adapter_name, target, kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/model.py", line 295, in _create_new_module new_module = dispatcher(target, adapter_name, lora_config=lora_config, kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 1056, in dispatch_default new_module = Linear(target, adapter_name, **kwargs) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 356, in init self.update_layer( File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 126, in update_layer self.dora_init(adapter_name) File "/opt/conda/lib/python3.8/site-packages/peft/tuners/lora/layer.py", line 191, in dora_init lora_weight = lora_B.weight @ lora_A.weight RuntimeError: "addmm_implcpu" not implemented for 'Half'

Expected behavior

切换到 LLaMA-Factory_240321 的版本就没有这个问题

System Info

absl-py 1.2.0 accelerate 0.27.2 aiofiles 23.2.1 aiohttp 3.9.3 aiosignal 1.3.1 aip-model-repo 2.0.11 alabaster 0.7.12 altair 5.2.0 annotated-types 0.6.0 anthropic 0.21.3 anyio 4.2.0 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.0.8 async-timeout 4.0.3 attrs 23.2.0 audioread 3.0.0 Babel 2.10.3 backcall 0.2.0 backports.functools-lru-cache 1.6.4 backports.zoneinfo 0.2.1 beautifulsoup4 4.11.1 bitsandbytes 0.41.3 bleach 5.0.1 blessed 1.20.0 blis 0.7.8 brotlipy 0.7.0 cachetools 5.2.0 catalogue 2.0.8 certifi 2022.9.14 cffi 1.15.1 chardet 5.0.0 charset-normalizer 2.1.0 click 8.1.3 cloudpickle 2.2.0 codecov 2.1.12 colorama 0.4.5 comm 0.2.0 conda 4.14.0 conda-build 3.22.0 conda-package-handling 1.9.0 contourpy 1.0.5 coverage 6.4.4 cryptography 37.0.4 cuda-python 11.7.0+0.g95a2041.dirty cudf 22.8.0a0+304.g6ca81bbc78.dirty cugraph 22.8.0a0+132.g2daa31b6.dirty cuml 22.8.0a0+52.g73b8d00d0.dirty cupy-cuda118 11.0.0 cycler 0.11.0 cymem 2.0.6 Cython 0.29.32 dask 2022.7.1 dask-cuda 22.8.0a0+36.g9860cad dask-cudf 22.8.0a0+304.g6ca81bbc78.dirty dataclasses 0.8 datasets 2.17.1 debugpy 1.6.3 decorator 5.1.1 deepspeed 0.13.4 defusedxml 0.7.1 dill 0.3.8 distributed 2022.7.1 distro 1.9.0 docstring_parser 0.16 docutils 0.17.1 einops 0.7.0 emoji 2.11.0 entrypoints 0.3 exceptiongroup 1.2.0 executing 1.0.0 expecttest 0.1.3 fastapi 0.110.0 fastjsonschema 2.16.2 fastrlock 0.8 ffmpy 0.3.2 filelock 3.8.0 fire 0.6.0 flake8 3.7.9 flash-attn 2.5.5 Flask 2.2.2 fonttools 4.37.3 fqdn 1.5.1 frozenlist 1.4.1 fschat 0.2.36 fsspec 2023.10.0 functorch 0.3.0a0 future 0.18.2 glob2 0.7 google-auth 2.11.1 google-auth-oauthlib 0.4.6 gpustat 1.1.1 gradio 3.50.2 gradio_client 0.6.1 graphsurgeon 0.4.6 grpcio 1.49.1 h11 0.14.0 HeapDict 1.0.1 hjson 3.1.0 httpcore 1.0.5 httpx 0.27.0 huggingface-hub 0.22.1 hypothesis 4.50.8 idna 3.3 imagesize 1.4.1 importlib-metadata 4.12.0 importlib-resources 5.9.0 iniconfig 1.1.1 iopath 0.1.10 ipykernel 6.15.3 ipython 8.5.0 ipython-genutils 0.2.0 ipywidgets 8.1.1 isoduration 20.11.0 itsdangerous 2.1.2 jedi 0.18.1 jieba 0.42.1 Jinja2 3.1.2 joblib 1.2.0 json5 0.9.10 jsonpointer 2.4 jsonschema 4.20.0 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter_client 8.6.0 jupyter-console 6.6.3 jupyter_core 5.6.1 jupyter-events 0.9.0 jupyter_server 2.12.1 jupyter_server_terminals 0.5.1 jupyter-tensorboard 0.2.0 jupyterlab 2.3.2 jupyterlab-pygments 0.2.2 jupyterlab-server 1.2.0 jupyterlab-widgets 3.0.9 jupytext 1.14.1 kiwisolver 1.4.4 langcodes 3.3.0 Levenshtein 0.25.1 libarchive-c 4.0 librosa 0.9.2 llvmlite 0.39.1 lmdb 1.3.0 locket 1.0.0 lxml 4.9.1 Markdown 3.4.1 markdown-it-py 3.0.0 markdown2 2.4.13 MarkupSafe 2.1.1 matplotlib 3.6.0 matplotlib-inline 0.1.6 mccabe 0.6.1 mdit-py-plugins 0.3.0 mdurl 0.1.2 mistune 2.0.4 mock 4.0.3 mpmath 1.3.0 msgpack 1.0.4 multidict 6.0.5 multiprocess 0.70.16 murmurhash 1.0.6 nbclient 0.6.8 nbconvert 7.0.0 nbformat 5.6.0 nest-asyncio 1.5.5 networkx 2.6.3 nh3 0.2.17 ninja 1.11.1.1 nltk 3.7 notebook 6.4.10 numba 0.56.2 numpy 1.22.2 nvidia-dali-cuda110 1.17.0 nvidia-ml-py 12.535.133 nvidia-pyindex 1.0.9 nvtx 0.2.5 oauthlib 3.2.1 onnx 1.12.0 openai 0.27.8 orjson 3.10.0 overrides 7.4.0 packaging 21.3 pandas 1.4.4 pandocfilters 1.5.0 parso 0.8.3 partd 1.3.0 pathy 0.6.2 peft 0.10.0 pexpect 4.8.0 pickleshare 0.7.5 Pillow 9.0.1 pip 23.3.2 pkginfo 1.8.3 pkgutil_resolve_name 1.3.10 platformdirs 4.1.0 pluggy 1.0.0 polygraphy 0.38.0 pooch 1.6.0 portalocker 2.5.1 preshed 3.0.6 prettytable 3.4.1 prometheus-client 0.14.1 prompt-toolkit 3.0.31 protobuf 3.20.2 psutil 5.9.2 ptyprocess 0.7.0 pure-eval 0.2.2 py 1.11.0 py-cpuinfo 9.0.0 pyarrow 15.0.2 pyarrow-hotfix 0.6 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybind11 2.10.0 pycocotools 2.0+nv0.6.0 pycodestyle 2.5.0 pycosat 0.6.3 pycparser 2.21 pycrypto 2.6.1 pycryptodomex 3.19.1 pydantic 2.6.4 pydantic_core 2.16.3 pydot 1.4.2 pydub 0.25.1 pyflakes 2.1.1 Pygments 2.13.0 pylibcugraph 22.8.0a0+132.g2daa31b6.dirty PyMySQL 1.1.0 pynvml 11.4.1 pyOpenSSL 22.0.0 pyparsing 3.0.9 pyrsistent 0.18.1 PySocks 1.7.1 pytest 6.2.5 pytest-cov 3.0.0 pytest-pythonpath 0.7.4 pythainlp 5.0.2 python-dateutil 2.8.2 python-hostlist 1.21 python-json-logger 2.0.7 python-Levenshtein 0.25.1 python-multipart 0.0.9 python-nvd3 0.15.0 python-slugify 6.1.2 pytorch-quantization 2.1.2 pytz 2022.2.1 PyYAML 6.0 pyzmq 24.0.1 qtconsole 5.5.1 QtPy 2.4.1 raft 22.8.0a0+70.g9070c30.dirty rapidfuzz 3.8.1 ray 2.10.0 referencing 0.32.0 regex 2022.9.13 requests 2.28.1 requests-oauthlib 1.3.1 resampy 0.4.2 revtok 0.0.3 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.7.1 rmm 22.8.0a0+62.gf6bf047.dirty rouge-chinese 1.0.3 rpds-py 0.16.2 rsa 4.9 ruamel-yaml-conda 0.15.80 sacremoses 0.0.53 safetensors 0.4.2 scikit-learn 0.24.2 scipy 1.6.3 semantic-version 2.10.0 Send2Trash 1.8.2 sentence-transformers 2.6.1 sentencepiece 0.2.0 setuptools 59.5.0 shellingham 1.5.0 shopee-aip-dataset-mount 0.591 shopee-chatbot-text-clustering 1.0.0 shortuuid 1.0.13 shtab 1.7.1 six 1.16.0 smart-open 5.2.1 sniffio 1.3.0 snowballstemmer 2.2.0 sortedcontainers 2.4.0 SoundFile 0.10.3.post1 soupsieve 2.3.2.post1 spacy 3.2.2 spacy-legacy 3.0.10 spacy-loggers 1.0.3 Sphinx 5.1.1 sphinx-glpi-theme 0.3 sphinx-rtd-theme 1.0.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 srsly 2.4.2 sse-starlette 2.0.0 stack-data 0.5.0 starlette 0.36.3 svgwrite 1.4.3 sympy 1.12 tabulate 0.8.10 tblib 1.7.0 tenacity 8.2.3 tensorboard 2.10.0 tensorboard-data-server 0.6.1 tensorboard-plugin-wit 1.8.1 tensorrt 8.5.0.12 termcolor 2.4.0 terminado 0.15.0 text-unidecode 1.3 thinc 8.0.13 threadpoolctl 3.1.0 tiktoken 0.6.0 tinycss2 1.1.1 tokenizers 0.19.1 toml 0.10.2 tomli 2.0.1 toolz 0.12.0 torch 2.1.2+cu118 torch-tensorrt 1.2.0a0 torchaudio 2.1.2+cu118 torchtext 0.11.0a0 torchvision 0.16.2+cu118 tornado 6.4 tqdm 4.64.1 traitlets 5.9.0 transformers 4.40.0 treelite 2.4.0 treelite-runtime 2.4.0 triton 2.1.0 trl 0.8.1 typer 0.4.2 types-python-dateutil 2.8.19.14 typing_extensions 4.10.0 tyro 0.7.3 ucx-py 0.27.0a0+29.ge9e81f8 uff 0.6.9 uri-template 1.3.0 urllib3 1.26.11 uvicorn 0.27.1 wasabi 0.10.0 wavedrom 2.0.3.post3 wcwidth 0.2.5 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 websockets 11.0.3 Werkzeug 2.2.2 wheel 0.37.1 widgetsnbextension 4.0.9 xgboost 1.6.1 xxhash 3.4.1 yarl 1.9.4 zict 2.2.0 zipp 3.8.1

Others

No response

hiyouga commented 6 months ago

--export_device cuda