Yi-34B-Chat-4bits运行报错

wells-Qiang-Chen commented 9 months ago

Reminder

[X] I have searched the Github Discussion and issues and have not found anything similar to this.

Environment

- OS:Ubuntu 18.04
- Python:3.10
- PyTorch:2.0.1
- CUDA:release 11.8, V11.8.89

Current Behavior

File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decoratecontext return func(*args, **kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/awq/modules/fused/model.py", line 39, in forward h, , past_key_value = layer(h, None, attention_mask=mask, is_causal=is_causal) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/awq/modules/fused/block.py", line 27, in forward attnoutput, , past_key_value = self.attn.forward( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/awq/modules/fused/attn.py", line 141, in forward xqkv = self.qkv_proj(hidden_states) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/awq/modules/linear.py", line 105, in forward out = awq_inference_engine.gemm_forward_cuda(x.reshape(-1, x.shape[-1]), self.qweight, self.scales, self.qzeros, 8) RuntimeError: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. 加入环境变量CUDA_LAUNCH_BLOCKING=1.后： output_ids = model.generate(input_ids.to('cuda:0')) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/generation/utils.py", line 1764, in generate return self.sample( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/generation/utils.py", line 2861, in sample outputs = self( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1181, in forward outputs = self.model( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1068, in forward layer_outputs = decoder_layer( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 796, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 386, in forward query_states = self.q_proj(hidden_states) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "/home/data/cqiang/Anaconda3/envs/yi/lib/python3.10/site-packages/awq/modules/linear.py", line 105, in forward out = awq_inference_engine.gemm_forward_cuda(x.reshape(-1, x.shape[-1]), self.qweight, self.scales, self.qzeros, 8) RuntimeError: CUDA error: no kernel image is available for execution on the device Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Expected Behavior

No response

Steps to Reproduce

使用的是v100 32g，cuda 11.8 环境为： _libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main _openmp_mutex 5.1 1_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main absl-py 2.1.0 pypi_0 pypi accelerate 0.26.1 pypi_0 pypi aiofiles 23.2.1 pypi_0 pypi aiohttp 3.9.1 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi altair 5.2.0 pypi_0 pypi annotated-types 0.6.0 pypi_0 pypi anyio 4.2.0 pypi_0 pypi argon2-cffi 21.3.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main argon2-cffi-bindings 21.2.0 pypi_0 pypi asttokens 2.0.5 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main async-lru 2.0.4 pypi_0 pypi async-timeout 4.0.3 pypi_0 pypi attributedict 0.3.0 pypi_0 pypi attrs 23.1.0 pypi_0 pypi autoawq 0.1.6+cu118 pypi_0 pypi babel 2.11.0 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free bleach 4.1.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main blessings 1.7 pypi_0 pypi brotli 1.0.9 pypi_0 pypi brotli-python 1.0.9 py310h6a678d5_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bzip2 1.0.8 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ca-certificates 2023.12.12 h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cachetools 5.3.2 pypi_0 pypi certifi 2023.11.17 pypi_0 pypi cffi 1.16.0 pypi_0 pypi chardet 5.2.0 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi click 8.1.7 pypi_0 pypi cmake 3.28.1 pypi_0 pypi codecov 2.1.13 pypi_0 pypi colorama 0.4.6 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi colour-runner 0.1.1 pypi_0 pypi comm 0.1.2 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi coverage 7.4.0 pypi_0 pypi cryptography 41.0.7 pypi_0 pypi cuda-cudart 11.8.89 0 nvidia cuda-cupti 11.8.87 0 nvidia cuda-libraries 11.8.0 0 nvidia cuda-nvrtc 11.8.89 0 nvidia cuda-nvtx 11.8.86 0 nvidia cuda-runtime 11.8.0 0 nvidia cycler 0.12.1 pypi_0 pypi cyrus-sasl 2.1.28 h52b45da_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main dataproperty 1.0.1 pypi_0 pypi datasets 2.16.1 pypi_0 pypi dbus 1.13.18 hb2f20db_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main debugpy 1.6.7 pypi_0 pypi decorator 5.1.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main deepdiff 6.7.1 pypi_0 pypi defusedxml 0.7.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main dill 0.3.7 pypi_0 pypi distlib 0.3.8 pypi_0 pypi evaluate 0.4.1 pypi_0 pypi exceptiongroup 1.2.0 pypi_0 pypi executing 0.8.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main expat 2.5.0 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main fastapi 0.109.0 pypi_0 pypi fastjsonschema 2.16.2 pypi_0 pypi ffmpeg 4.3 hf484d3e_0 pytorch ffmpy 0.3.1 pypi_0 pypi filelock 3.13.1 pypi_0 pypi fontconfig 2.14.1 h4c34cd2_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main fonttools 4.47.2 pypi_0 pypi freetype 2.12.1 h4a9f257_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main frozenlist 1.4.1 pypi_0 pypi fsspec 2023.10.0 pypi_0 pypi giflib 5.2.1 h5eee18b_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main glib 2.69.1 he621ea3_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gmp 6.2.1 h295c915_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gmpy2 2.1.2 pypi_0 pypi gnutls 3.6.15 he1e5248_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gradio 4.15.0 pypi_0 pypi gradio-client 0.8.1 pypi_0 pypi gst-plugins-base 1.14.1 h6a678d5_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gstreamer 1.14.1 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main h11 0.14.0 pypi_0 pypi httpcore 1.0.2 pypi_0 pypi httpx 0.26.0 pypi_0 pypi huggingface-hub 0.20.3 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi icu 73.1 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main idna 3.4 pypi_0 pypi importlib-resources 6.1.1 pypi_0 pypi inspecta 0.1.3 pypi_0 pypi intel-openmp 2023.1.0 hdb19cb5_46306 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ipykernel 6.28.0 pypi_0 pypi ipython 8.20.0 pypi_0 pypi ipywidgets 8.0.4 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jinja2 3.1.3 pypi_0 pypi joblib 1.3.2 pypi_0 pypi jpeg 9e h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main json5 0.9.6 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jsonlines 4.0.0 pypi_0 pypi jsonschema 4.19.2 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 8.6.0 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-core 5.5.0 pypi_0 pypi jupyter-events 0.8.0 pypi_0 pypi jupyter-lsp 2.2.0 pypi_0 pypi jupyter-server 2.10.0 pypi_0 pypi jupyter-server-terminals 0.4.4 pypi_0 pypi jupyter_client 8.6.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyter_console 6.6.3 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyter_core 5.5.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyter_events 0.8.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyter_server 2.10.0 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyter_server_terminals 0.4.4 py310h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyterlab 4.0.8 pypi_0 pypi jupyterlab-server 2.25.1 pypi_0 pypi jupyterlab-widgets 3.0.9 pypi_0 pypi jupyterlab_pygments 0.1.2 py_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyterlab_server 2.25.1 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jupyterlab_widgets 3.0.9 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main kiwisolver 1.4.5 pypi_0 pypi krb5 1.20.1 h143b758_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lame 3.100 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lcms2 2.12 h3be6417_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ld_impl_linux-64 2.38 h1181459_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lerc 3.0 h295c915_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libclang 14.0.6 default_hc6dbbc7_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libclang13 14.0.6 default_he11475f_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libcublas 11.11.3.6 0 nvidia libcufft 10.9.0.58 0 nvidia libcufile 1.8.1.2 0 nvidia libcups 2.4.2 h2d74bed_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libcurand 10.3.4.107 0 nvidia libcusolver 11.4.1.48 0 nvidia libcusparse 11.7.5.86 0 nvidia libdeflate 1.17 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libedit 3.1.20230828 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libffi 3.4.4 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgcc-ng 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgomp 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libiconv 1.14 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free libidn2 2.3.4 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libllvm14 14.0.6 hdb19cb5_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libnpp 11.8.0.86 0 nvidia libnvjpeg 11.9.0.86 0 nvidia libpng 1.6.39 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libpq 12.15 hdbd6064_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libsodium 1.0.18 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-ng 11.2.0 h1234567_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libtasn1 4.19.0 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libtiff 4.5.1 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libunistring 0.9.10 h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libuuid 1.41.5 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libwebp 1.3.2 h11a3e52_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libwebp-base 1.3.2 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libxcb 1.15 h7f8727e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libxkbcommon 1.0.1 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libxml2 2.10.4 hf1b16e4_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lit 17.0.6 pypi_0 pypi lm-eval 0.4.0 pypi_0 pypi lxml 5.1.0 pypi_0 pypi lz4-c 1.9.4 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.4 pypi_0 pypi matplotlib 3.8.2 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mbstrdecoder 1.1.3 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mistune 2.0.4 pypi_0 pypi mkl 2023.1.0 h213fc3f_46344 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl-fft 1.3.8 pypi_0 pypi mkl-random 1.2.4 pypi_0 pypi mkl-service 2.4.0 pypi_0 pypi mkl_fft 1.3.8 py310h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random 1.2.4 py310hdb19cb5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mpc 1.0.3 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free mpfr 4.0.2 hb69a4c5_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mpmath 1.3.0 pypi_0 pypi multidict 6.0.4 pypi_0 pypi multiprocess 0.70.15 pypi_0 pypi mysql 5.7.24 h721c034_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nbclient 0.8.0 pypi_0 pypi nbconvert 7.10.0 pypi_0 pypi nbformat 5.9.2 pypi_0 pypi ncurses 6.4 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.6 pypi_0 pypi nettle 3.7.3 hbbd107a_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main networkx 3.1 pypi_0 pypi nltk 3.8.1 pypi_0 pypi notebook 7.0.6 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numexpr 2.8.8 pypi_0 pypi numpy 1.26.3 pypi_0 pypi numpy-base 1.26.3 py310hb5e798b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nvidia-cublas-cu11 11.10.3.66 pypi_0 pypi nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi nvidia-cuda-cupti-cu11 11.7.101 pypi_0 pypi nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi nvidia-cuda-nvrtc-cu11 11.7.99 pypi_0 pypi nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi nvidia-cuda-runtime-cu11 11.7.99 pypi_0 pypi nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi nvidia-cudnn-cu11 8.5.0.96 pypi_0 pypi nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi nvidia-curand-cu11 10.2.10.91 pypi_0 pypi nvidia-curand-cu12 10.3.2.106 pypi_0 pypi nvidia-cusolver-cu11 11.4.0.1 pypi_0 pypi nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi nvidia-cusparse-cu11 11.7.4.91 pypi_0 pypi nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi nvidia-nccl-cu11 2.14.3 pypi_0 pypi nvidia-nccl-cu12 2.18.1 pypi_0 pypi nvidia-nvjitlink-cu12 12.3.101 pypi_0 pypi nvidia-nvtx-cu11 11.7.91 pypi_0 pypi nvidia-nvtx-cu12 12.1.105 pypi_0 pypi openh264 2.1.1 h4ff587b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main openjpeg 2.4.0 h3ad879b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main openssl 3.0.12 h7f8727e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ordered-set 4.1.0 pypi_0 pypi orjson 3.9.12 pypi_0 pypi overrides 7.4.0 pypi_0 pypi packaging 23.2 pypi_0 pypi pandas 2.2.0 pypi_0 pypi pandocfilters 1.5.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main parso 0.8.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pathvalidate 3.2.0 pypi_0 pypi pcre 8.45 h295c915_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main peft 0.7.1 pypi_0 pypi pexpect 4.8.0 pyhd3eb1b0_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pillow 10.2.0 pypi_0 pypi pip 23.3.1 pypi_0 pypi platformdirs 4.1.0 pypi_0 pypi pluggy 1.3.0 pypi_0 pypi ply 3.11 pypi_0 pypi portalocker 2.8.2 pypi_0 pypi prometheus-client 0.14.1 pypi_0 pypi prometheus_client 0.14.1 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main prompt-toolkit 3.0.43 pypi_0 pypi prompt_toolkit 3.0.43 hd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main protobuf 4.25.2 pypi_0 pypi psutil 5.9.8 pypi_0 pypi ptyprocess 0.7.0 pyhd3eb1b0_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pure_eval 0.2.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyarrow 15.0.0 pypi_0 pypi pyarrow-hotfix 0.6 pypi_0 pypi pybind11 2.11.1 pypi_0 pypi pycparser 2.21 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pydantic 2.5.3 pypi_0 pypi pydantic-core 2.14.6 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pygments 2.17.2 pypi_0 pypi pyopenssl 23.2.0 pypi_0 pypi pyparsing 3.1.1 pypi_0 pypi pyproject-api 1.6.1 pypi_0 pypi pyqt 5.15.10 py310h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyqt5 5.15.10 pypi_0 pypi pyqt5-sip 12.13.0 pypi_0 pypi pysocks 1.7.1 pypi_0 pypi pytablewriter 1.2.0 pypi_0 pypi python 3.10.13 h955ad1f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python-dateutil 2.8.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python-fastjsonschema 2.16.2 py310h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python-json-logger 2.0.7 pypi_0 pypi python-multipart 0.0.6 pypi_0 pypi pytorch 2.0.1 py3.10_cuda11.8_cudnn8.7.0_0 pytorch pytorch-cuda 11.8 h7e8668a_5 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2023.3.post1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi pyzmq 25.1.2 pypi_0 pypi qt-main 5.15.2 h53bd1ea_10 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main qtconsole 5.5.0 pypi_0 pypi qtpy 2.4.1 pypi_0 pypi readline 8.2 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main referencing 0.30.2 pypi_0 pypi regex 2023.12.25 pypi_0 pypi requests 2.31.0 pypi_0 pypi responses 0.18.0 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi rich 13.7.0 pypi_0 pypi rootpath 0.1.1 pypi_0 pypi rouge-score 0.1.2 pypi_0 pypi rpds-py 0.17.1 pypi_0 pypi ruff 0.1.14 pypi_0 pypi sacrebleu 2.4.0 pypi_0 pypi safetensors 0.4.1 pypi_0 pypi scikit-learn 1.4.0 pypi_0 pypi scipy 1.12.0 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi sentencepiece 0.1.99 pypi_0 pypi setuptools 68.2.2 pypi_0 pypi shellingham 1.5.4 pypi_0 pypi sip 6.7.12 pypi_0 pypi six 1.16.0 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sniffio 1.3.0 pypi_0 pypi soupsieve 2.5 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sqlitedict 2.1.0 pypi_0 pypi stack_data 0.2.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main starlette 0.35.1 pypi_0 pypi sympy 1.12 pypi_0 pypi tabledata 1.3.3 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tbb 2021.8.0 hdb19cb5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tcolorpy 0.1.4 pypi_0 pypi termcolor 2.4.0 pypi_0 pypi terminado 0.17.1 pypi_0 pypi texttable 1.7.0 pypi_0 pypi threadpoolctl 3.2.0 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tokenizers 0.15.1 pypi_0 pypi toml 0.10.2 pypi_0 pypi tomli 2.0.1 pypi_0 pypi tomlkit 0.12.0 pypi_0 pypi toolz 0.12.0 pypi_0 pypi torch 2.0.1 pypi_0 pypi torchaudio 2.0.2 pypi_0 pypi torchtriton 2.0.0 py310 pytorch torchvision 0.16.2 pypi_0 pypi tornado 6.3.3 pypi_0 pypi tox 4.12.1 pypi_0 pypi tqdm 4.66.1 pypi_0 pypi tqdm-multiprocess 0.0.11 pypi_0 pypi traitlets 5.7.1 pypi_0 pypi transformers 4.36.2 pypi_0 pypi triton 2.0.0 pypi_0 pypi typepy 1.3.2 pypi_0 pypi typer 0.9.0 pypi_0 pypi typing-extensions 4.9.0 pypi_0 pypi typing_extensions 4.9.0 py310h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tzdata 2023.4 pypi_0 pypi urllib3 2.1.0 pypi_0 pypi uvicorn 0.27.0 pypi_0 pypi virtualenv 20.25.0 pypi_0 pypi wcwidth 0.2.5 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main webencodings 0.5.1 pypi_0 pypi websocket-client 0.58.0 pypi_0 pypi websockets 11.0.3 pypi_0 pypi wheel 0.41.2 pypi_0 pypi widgetsnbextension 4.0.5 pypi_0 pypi xxhash 3.4.1 pypi_0 pypi xz 5.4.5 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main yaml 0.2.5 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main yarl 1.9.4 pypi_0 pypi zeromq 4.3.5 h6a678d5_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zlib 1.2.13 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zstandard 0.22.0 pypi_0 pypi zstd 1.5.5 hc292b87_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

Anything Else?

No response

Yimi81 commented 9 months ago

autoawq不支持v100, https://github.com/casper-hansen/AutoAWQ/issues/290

wells-Qiang-Chen commented 9 months ago

考虑出个gptq量化的int4模型吗

Yimi81 commented 9 months ago

这里我们推荐了第三方的各种格式的量化版本 https://github.com/01-ai/Yi?tab=readme-ov-file#%EF%B8%8F-quantitation

01-ai / Yi