search

01-ai / Yi

A series of large language models trained from scratch by developers @01-ai
https://01.ai
Apache License 2.0
7.6k stars 469 forks source link

6b运行正常，34b-int4运行失败 #266

Closed jrd77 closed 8 months ago

jrd77 commented 9 months ago

6B模型运行成功，Yi-34B-Chat-4bits运行失败。

QQ截图20231220230139

执行代码：


model_path = '/data1/llm/model/01ai/Yi-34B-Chat-4bits'

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False)

# Since transformers 4.35.0, the GPT-Q/AWQ model can be loaded using AutoModelForCausalLM.
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype='auto'
).eval()

# Prompt content: "hi"
messages = [
    {"role": "user", "content": "hi"}
]

input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')
output_ids = model.generate(input_ids.to('cuda'))
response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)

# Model response: "Hello! How can I assist you today?"
print(response)

报错信息：

RuntimeError: CUDA error: no kernel image is available for execution on the device

CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

运行环境：

ubuntu 22，NVIDIA V100 32G，cuda11.8

micromamba list



_libgcc_mutex         0.1           conda_forge                   conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2            
  _openmp_mutex         4.5           2_kmp_llvm                    conda-forge/linux-64/_openmp_mutex-4.5-2_kmp_llvm.tar.bz2             
  accelerate            0.24.1        pyhd8ed1ab_0                  conda-forge/noarch/accelerate-0.24.1-pyhd8ed1ab_0.conda               
  aiohttp               3.9.1         py310h2372a71_0               conda-forge/linux-64/aiohttp-3.9.1-py310h2372a71_0.conda              
  aiosignal             1.3.1         pyhd8ed1ab_0                  conda-forge/noarch/aiosignal-1.3.1-pyhd8ed1ab_0.tar.bz2               
  annotated-types       0.6.0         pyhd8ed1ab_0                  conda-forge/noarch/annotated-types-0.6.0-pyhd8ed1ab_0.conda           
  async-timeout         4.0.3         pyhd8ed1ab_0                  conda-forge/noarch/async-timeout-4.0.3-pyhd8ed1ab_0.conda             
  attrs                 23.1.0        pyh71513ae_1                  conda-forge/noarch/attrs-23.1.0-pyh71513ae_1.conda                    
  aws-c-auth            0.7.8         h5c941e0_1                    conda-forge/linux-64/aws-c-auth-0.7.8-h5c941e0_1.conda                
  aws-c-cal             0.6.9         h5d48c4d_2                    conda-forge/linux-64/aws-c-cal-0.6.9-h5d48c4d_2.conda                 
  aws-c-common          0.9.10        hd590300_0                    conda-forge/linux-64/aws-c-common-0.9.10-hd590300_0.conda             
  aws-c-compression     0.2.17        h7f92143_7                    conda-forge/linux-64/aws-c-compression-0.2.17-h7f92143_7.conda        
  aws-c-event-stream    0.3.2         h0bcb0bb_8                    conda-forge/linux-64/aws-c-event-stream-0.3.2-h0bcb0bb_8.conda        
  aws-c-http            0.7.14        hd268abd_3                    conda-forge/linux-64/aws-c-http-0.7.14-hd268abd_3.conda               
  aws-c-io              0.13.36       he14a76f_1                    conda-forge/linux-64/aws-c-io-0.13.36-he14a76f_1.conda                
  aws-c-mqtt            0.9.10        h35285c7_2                    conda-forge/linux-64/aws-c-mqtt-0.9.10-h35285c7_2.conda               
  aws-c-s3              0.4.3         h0448019_0                    conda-forge/linux-64/aws-c-s3-0.4.3-h0448019_0.conda                  
  aws-c-sdkutils        0.1.12        h7f92143_6                    conda-forge/linux-64/aws-c-sdkutils-0.1.12-h7f92143_6.conda           
  aws-checksums         0.1.17        h7f92143_6                    conda-forge/linux-64/aws-checksums-0.1.17-h7f92143_6.conda            
  aws-crt-cpp           0.24.9        h4a91382_1                    conda-forge/linux-64/aws-crt-cpp-0.24.9-h4a91382_1.conda              
  aws-sdk-cpp           1.11.210      h6d06844_3                    conda-forge/linux-64/aws-sdk-cpp-1.11.210-h6d06844_3.conda            
  blas                  2.120         mkl                           conda-forge/linux-64/blas-2.120-mkl.conda                             
  blas-devel            3.9.0         20_linux64_mkl                conda-forge/linux-64/blas-devel-3.9.0-20_linux64_mkl.conda            
  brotli-python         1.1.0         py310hc6cd4ac_1               conda-forge/linux-64/brotli-python-1.1.0-py310hc6cd4ac_1.conda        
  bzip2                 1.0.8         hd590300_5                    conda-forge/linux-64/bzip2-1.0.8-hd590300_5.conda                     
  c-ares                1.23.0        hd590300_0                    conda-forge/linux-64/c-ares-1.23.0-hd590300_0.conda                   
  ca-certificates       2023.11.17    hbcca054_0                    conda-forge/linux-64/ca-certificates-2023.11.17-hbcca054_0.conda      
  certifi               2023.11.17    pyhd8ed1ab_0                  conda-forge/noarch/certifi-2023.11.17-pyhd8ed1ab_0.conda              
  charset-normalizer    3.3.2         pyhd8ed1ab_0                  conda-forge/noarch/charset-normalizer-3.3.2-pyhd8ed1ab_0.conda        
  click                 8.1.7         unix_pyh707e725_0             conda-forge/noarch/click-8.1.7-unix_pyh707e725_0.conda                
  colorama              0.4.6         pyhd8ed1ab_0                  conda-forge/noarch/colorama-0.4.6-pyhd8ed1ab_0.tar.bz2                
  coloredlogs           15.0.1        pyhd8ed1ab_3                  conda-forge/noarch/coloredlogs-15.0.1-pyhd8ed1ab_3.tar.bz2            
  cuda-cudart           11.8.89       0                             nvidia/linux-64/cuda-cudart-11.8.89-0.tar.bz2                         
  cuda-cupti            11.8.87       0                             nvidia/linux-64/cuda-cupti-11.8.87-0.tar.bz2                          
  cuda-libraries        11.8.0        0                             nvidia/linux-64/cuda-libraries-11.8.0-0.tar.bz2                       
  cuda-nvrtc            11.8.89       0                             nvidia/linux-64/cuda-nvrtc-11.8.89-0.tar.bz2                          
  cuda-nvtx             11.8.86       0                             nvidia/linux-64/cuda-nvtx-11.8.86-0.tar.bz2                           
  cuda-runtime          11.8.0        0                             nvidia/linux-64/cuda-runtime-11.8.0-0.tar.bz2                         
  dataclasses           0.8           pyhc8e2a94_3                  conda-forge/noarch/dataclasses-0.8-pyhc8e2a94_3.tar.bz2               
  datasets              2.14.5        pyhd8ed1ab_0                  conda-forge/noarch/datasets-2.14.5-pyhd8ed1ab_0.conda                 
  deepspeed             0.12.2        cpu_py310h11dbdba_1           conda-forge/linux-64/deepspeed-0.12.2-cpu_py310h11dbdba_1.conda       
  dill                  0.3.7         pyhd8ed1ab_0                  conda-forge/noarch/dill-0.3.7-pyhd8ed1ab_0.conda                      
  einops                0.7.0         pyhd8ed1ab_1                  conda-forge/noarch/einops-0.7.0-pyhd8ed1ab_1.conda                    
  filelock              3.13.1        pyhd8ed1ab_0                  conda-forge/noarch/filelock-3.13.1-pyhd8ed1ab_0.conda                 
  frozenlist            1.4.0         py310h2372a71_1               conda-forge/linux-64/frozenlist-1.4.0-py310h2372a71_1.conda           
  fsspec                2023.6.0      pyh1a96a4e_0                  conda-forge/noarch/fsspec-2023.6.0-pyh1a96a4e_0.conda                 
  gflags                2.2.2         he1b5a44_1004                 conda-forge/linux-64/gflags-2.2.2-he1b5a44_1004.tar.bz2               
  glog                  0.6.0         h6f12383_0                    conda-forge/linux-64/glog-0.6.0-h6f12383_0.tar.bz2                    
  gmp                   6.3.0         h59595ed_0                    conda-forge/linux-64/gmp-6.3.0-h59595ed_0.conda                       
  gmpy2                 2.1.2         py310h3ec546c_1               conda-forge/linux-64/gmpy2-2.1.2-py310h3ec546c_1.tar.bz2              
  hjson-py              3.1.0         pyhd8ed1ab_0                  conda-forge/noarch/hjson-py-3.1.0-pyhd8ed1ab_0.tar.bz2                
  huggingface_hub       0.16.4        pyhd8ed1ab_0                  conda-forge/noarch/huggingface_hub-0.16.4-pyhd8ed1ab_0.conda          
  humanfriendly         10.0          pyhd8ed1ab_6                  conda-forge/noarch/humanfriendly-10.0-pyhd8ed1ab_6.conda              
  icu                   73.2          h59595ed_0                    conda-forge/linux-64/icu-73.2-h59595ed_0.conda                        
  idna                  3.6           pyhd8ed1ab_0                  conda-forge/noarch/idna-3.6-pyhd8ed1ab_0.conda                        
  importlib-metadata    6.9.0         pyha770c72_0                  conda-forge/noarch/importlib-metadata-6.9.0-pyha770c72_0.conda        
  importlib_metadata    6.9.0         hd8ed1ab_0                    conda-forge/noarch/importlib_metadata-6.9.0-hd8ed1ab_0.conda          
  jinja2                3.1.2         pyhd8ed1ab_1                  conda-forge/noarch/jinja2-3.1.2-pyhd8ed1ab_1.tar.bz2                  
  joblib                1.3.2         pyhd8ed1ab_0                  conda-forge/noarch/joblib-1.3.2-pyhd8ed1ab_0.conda                    
  keyutils              1.6.1         h166bdaf_0                    conda-forge/linux-64/keyutils-1.6.1-h166bdaf_0.tar.bz2                
  krb5                  1.21.2        h659d440_0                    conda-forge/linux-64/krb5-1.21.2-h659d440_0.conda                     
  ld_impl_linux-64      2.40          h41732ed_0                    conda-forge/linux-64/ld_impl_linux-64-2.40-h41732ed_0.conda           
  libabseil             20230802.1    cxx17_h59595ed_0              conda-forge/linux-64/libabseil-20230802.1-cxx17_h59595ed_0.conda      
  libaio                0.3.113       h166bdaf_0                    conda-forge/linux-64/libaio-0.3.113-h166bdaf_0.tar.bz2                
  libarrow              14.0.1        h422ced8_7_cpu                conda-forge/linux-64/libarrow-14.0.1-h422ced8_7_cpu.conda             
  libarrow-acero        14.0.1        h59595ed_7_cpu                conda-forge/linux-64/libarrow-acero-14.0.1-h59595ed_7_cpu.conda       
  libarrow-dataset      14.0.1        h59595ed_7_cpu                conda-forge/linux-64/libarrow-dataset-14.0.1-h59595ed_7_cpu.conda     
  libarrow-flight       14.0.1        h120cb0d_7_cpu                conda-forge/linux-64/libarrow-flight-14.0.1-h120cb0d_7_cpu.conda      
  libarrow-flight-sql   14.0.1        h61ff412_7_cpu                conda-forge/linux-64/libarrow-flight-sql-14.0.1-h61ff412_7_cpu.conda  
  libarrow-gandiva      14.0.1        hacb8726_7_cpu                conda-forge/linux-64/libarrow-gandiva-14.0.1-hacb8726_7_cpu.conda     
  libarrow-substrait    14.0.1        h61ff412_7_cpu                conda-forge/linux-64/libarrow-substrait-14.0.1-h61ff412_7_cpu.conda   
  libblas               3.9.0         20_linux64_mkl                conda-forge/linux-64/libblas-3.9.0-20_linux64_mkl.conda               
  libbrotlicommon       1.1.0         hd590300_1                    conda-forge/linux-64/libbrotlicommon-1.1.0-hd590300_1.conda           
  libbrotlidec          1.1.0         hd590300_1                    conda-forge/linux-64/libbrotlidec-1.1.0-hd590300_1.conda              
  libbrotlienc          1.1.0         hd590300_1                    conda-forge/linux-64/libbrotlienc-1.1.0-hd590300_1.conda              
  libcblas              3.9.0         20_linux64_mkl                conda-forge/linux-64/libcblas-3.9.0-20_linux64_mkl.conda              
  libcrc32c             1.1.2         h9c3ff4c_0                    conda-forge/linux-64/libcrc32c-1.1.2-h9c3ff4c_0.tar.bz2               
  libcublas             11.11.3.6     0                             nvidia/linux-64/libcublas-11.11.3.6-0.tar.bz2                         
  libcufft              10.9.0.58     0                             nvidia/linux-64/libcufft-10.9.0.58-0.tar.bz2                          
  libcufile             1.8.1.2       0                             nvidia/linux-64/libcufile-1.8.1.2-0.tar.bz2                           
  libcurand             10.3.4.101    0                             nvidia/linux-64/libcurand-10.3.4.101-0.tar.bz2                        
  libcurl               8.4.0         hca28451_0                    conda-forge/linux-64/libcurl-8.4.0-hca28451_0.conda                   
  libcusolver           11.4.1.48     0                             nvidia/linux-64/libcusolver-11.4.1.48-0.tar.bz2                       
  libcusparse           11.7.5.86     0                             nvidia/linux-64/libcusparse-11.7.5.86-0.tar.bz2                       
  libedit               3.1.20191231  he28a2e2_2                    conda-forge/linux-64/libedit-3.1.20191231-he28a2e2_2.tar.bz2          
  libev                 4.33          h516909a_1                    conda-forge/linux-64/libev-4.33-h516909a_1.tar.bz2                    
  libevent              2.1.12        hf998b51_1                    conda-forge/linux-64/libevent-2.1.12-hf998b51_1.conda                 
  libffi                3.4.2         h7f98852_5                    conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2                  
  libgcc-ng             13.2.0        h807b86a_3                    conda-forge/linux-64/libgcc-ng-13.2.0-h807b86a_3.conda                
  libgfortran-ng        13.2.0        h69a702a_3                    conda-forge/linux-64/libgfortran-ng-13.2.0-h69a702a_3.conda           
  libgfortran5          13.2.0        ha4646dd_3                    conda-forge/linux-64/libgfortran5-13.2.0-ha4646dd_3.conda             
  libgoogle-cloud       2.12.0        h5206363_4                    conda-forge/linux-64/libgoogle-cloud-2.12.0-h5206363_4.conda          
  libgrpc               1.59.3        hd6c4280_0                    conda-forge/linux-64/libgrpc-1.59.3-hd6c4280_0.conda                  
  libhwloc              2.9.3         default_h554bfaf_1009         conda-forge/linux-64/libhwloc-2.9.3-default_h554bfaf_1009.conda       
  libiconv              1.17          h166bdaf_0                    conda-forge/linux-64/libiconv-1.17-h166bdaf_0.tar.bz2                 
  liblapack             3.9.0         20_linux64_mkl                conda-forge/linux-64/liblapack-3.9.0-20_linux64_mkl.conda             
  liblapacke            3.9.0         20_linux64_mkl                conda-forge/linux-64/liblapacke-3.9.0-20_linux64_mkl.conda            
  libllvm15             15.0.7        h5cf9203_3                    conda-forge/linux-64/libllvm15-15.0.7-h5cf9203_3.conda                
  libnghttp2            1.58.0        h47da74e_0                    conda-forge/linux-64/libnghttp2-1.58.0-h47da74e_0.conda               
  libnpp                11.8.0.86     0                             nvidia/linux-64/libnpp-11.8.0.86-0.tar.bz2                            
  libnsl                2.0.1         hd590300_0                    conda-forge/linux-64/libnsl-2.0.1-hd590300_0.conda                    
  libnuma               2.0.16        h0b41bf4_1                    conda-forge/linux-64/libnuma-2.0.16-h0b41bf4_1.conda                  
  libnvjpeg             11.9.0.86     0                             nvidia/linux-64/libnvjpeg-11.9.0.86-0.tar.bz2                         
  libparquet            14.0.1        h352af49_7_cpu                conda-forge/linux-64/libparquet-14.0.1-h352af49_7_cpu.conda           
  libprotobuf           4.24.4        hf27288f_0                    conda-forge/linux-64/libprotobuf-4.24.4-hf27288f_0.conda              
  libre2-11             2023.06.02    h7a70373_0                    conda-forge/linux-64/libre2-11-2023.06.02-h7a70373_0.conda            
  libsentencepiece      0.1.99        h866249d_5                    conda-forge/linux-64/libsentencepiece-0.1.99-h866249d_5.conda         
  libsqlite             3.44.2        h2797004_0                    conda-forge/linux-64/libsqlite-3.44.2-h2797004_0.conda                
  libssh2               1.11.0        h0841786_0                    conda-forge/linux-64/libssh2-1.11.0-h0841786_0.conda                  
  libstdcxx-ng          13.2.0        h7e041cc_3                    conda-forge/linux-64/libstdcxx-ng-13.2.0-h7e041cc_3.conda             
  libthrift             0.19.0        hb90f79a_1                    conda-forge/linux-64/libthrift-0.19.0-hb90f79a_1.conda                
  libutf8proc           2.8.0         h166bdaf_0                    conda-forge/linux-64/libutf8proc-2.8.0-h166bdaf_0.tar.bz2             
  libuuid               2.38.1        h0b41bf4_0                    conda-forge/linux-64/libuuid-2.38.1-h0b41bf4_0.conda                  
  libxml2               2.11.6        h232c23b_0                    conda-forge/linux-64/libxml2-2.11.6-h232c23b_0.conda                  
  libzlib               1.2.13        hd590300_5                    conda-forge/linux-64/libzlib-1.2.13-hd590300_5.conda                  
  llvm-openmp           17.0.6        h4dfa4b3_0                    conda-forge/linux-64/llvm-openmp-17.0.6-h4dfa4b3_0.conda              
  lz4-c                 1.9.4         hcb278e6_0                    conda-forge/linux-64/lz4-c-1.9.4-hcb278e6_0.conda                     
  markupsafe            2.1.3         py310h2372a71_1               conda-forge/linux-64/markupsafe-2.1.3-py310h2372a71_1.conda           
  mkl                   2023.2.0      h84fe81f_50496                conda-forge/linux-64/mkl-2023.2.0-h84fe81f_50496.conda                
  mkl-devel             2023.2.0      ha770c72_50496                conda-forge/linux-64/mkl-devel-2023.2.0-ha770c72_50496.conda          
  mkl-include           2023.2.0      h84fe81f_50496                conda-forge/linux-64/mkl-include-2023.2.0-h84fe81f_50496.conda        
  mpc                   1.3.1         hfe3b2da_0                    conda-forge/linux-64/mpc-1.3.1-hfe3b2da_0.conda                       
  mpfr                  4.2.1         h9458935_0                    conda-forge/linux-64/mpfr-4.2.1-h9458935_0.conda                      
  mpmath                1.3.0         pyhd8ed1ab_0                  conda-forge/noarch/mpmath-1.3.0-pyhd8ed1ab_0.conda                    
  multidict             6.0.4         py310h2372a71_1               conda-forge/linux-64/multidict-6.0.4-py310h2372a71_1.conda            
  multiprocess          0.70.15       py310h2372a71_1               conda-forge/linux-64/multiprocess-0.70.15-py310h2372a71_1.conda       
  ncurses               6.4           h59595ed_2                    conda-forge/linux-64/ncurses-6.4-h59595ed_2.conda                     
  networkx              3.2.1         pyhd8ed1ab_0                  conda-forge/noarch/networkx-3.2.1-pyhd8ed1ab_0.conda                  
  numpy                 1.26.2        py310hb13e2d6_0               conda-forge/linux-64/numpy-1.26.2-py310hb13e2d6_0.conda               
  openssl               3.2.0         hd590300_1                    conda-forge/linux-64/openssl-3.2.0-hd590300_1.conda                   
  optimum               1.13.2        pyhd8ed1ab_0                  conda-forge/noarch/optimum-1.13.2-pyhd8ed1ab_0.conda                  
  orc                   1.9.2         h4b38347_0                    conda-forge/linux-64/orc-1.9.2-h4b38347_0.conda                       
  packaging             23.2          pyhd8ed1ab_0                  conda-forge/noarch/packaging-23.2-pyhd8ed1ab_0.conda                  
  pandas                2.1.3         py310hcc13569_0               conda-forge/linux-64/pandas-2.1.3-py310hcc13569_0.conda               
  pip                   23.3.1        pyhd8ed1ab_0                  conda-forge/noarch/pip-23.3.1-pyhd8ed1ab_0.conda                      
  protobuf              4.24.4        py310h620c231_0               conda-forge/linux-64/protobuf-4.24.4-py310h620c231_0.conda            
  psutil                5.9.5         py310h2372a71_1               conda-forge/linux-64/psutil-5.9.5-py310h2372a71_1.conda               
  py-cpuinfo            9.0.0         pyhd8ed1ab_0                  conda-forge/noarch/py-cpuinfo-9.0.0-pyhd8ed1ab_0.tar.bz2              
  pyarrow               14.0.1        py310hf9e7431_7_cpu           conda-forge/linux-64/pyarrow-14.0.1-py310hf9e7431_7_cpu.conda         
  pydantic              2.5.2         pyhd8ed1ab_0                  conda-forge/noarch/pydantic-2.5.2-pyhd8ed1ab_0.conda                  
  pydantic-core         2.14.5        py310hcb5633a_0               conda-forge/linux-64/pydantic-core-2.14.5-py310hcb5633a_0.conda       
  pynvml                11.5.0        pyhd8ed1ab_0                  conda-forge/noarch/pynvml-11.5.0-pyhd8ed1ab_0.conda                   
  pysocks               1.7.1         pyha2e5f31_6                  conda-forge/noarch/pysocks-1.7.1-pyha2e5f31_6.tar.bz2                 
  python                3.10.13       hd12c33a_0_cpython            conda-forge/linux-64/python-3.10.13-hd12c33a_0_cpython.conda          
  python-dateutil       2.8.2         pyhd8ed1ab_0                  conda-forge/noarch/python-dateutil-2.8.2-pyhd8ed1ab_0.tar.bz2         
  python-tzdata         2023.3        pyhd8ed1ab_0                  conda-forge/noarch/python-tzdata-2023.3-pyhd8ed1ab_0.conda            
  python-xxhash         3.4.1         py310h2372a71_0               conda-forge/linux-64/python-xxhash-3.4.1-py310h2372a71_0.conda        
  python_abi            3.10          4_cp310                       conda-forge/linux-64/python_abi-3.10-4_cp310.conda                    
  pytorch               2.0.1         py3.10_cuda11.8_cudnn8.7.0_0  pytorch/linux-64/pytorch-2.0.1-py3.10_cuda11.8_cudnn8.7.0_0.tar.bz2   
  pytorch-cuda          11.8          h7e8668a_5                    pytorch/linux-64/pytorch-cuda-11.8-h7e8668a_5.tar.bz2                 
  pytorch-mutex         1.0           cuda                          pytorch/noarch/pytorch-mutex-1.0-cuda.tar.bz2                         
  pytz                  2023.3.post1  pyhd8ed1ab_0                  conda-forge/noarch/pytz-2023.3.post1-pyhd8ed1ab_0.conda               
  pyyaml                6.0.1         py310h2372a71_1               conda-forge/linux-64/pyyaml-6.0.1-py310h2372a71_1.conda               
  rdma-core             49.0          hd3aeb46_1                    conda-forge/linux-64/rdma-core-49.0-hd3aeb46_1.conda                  
  re2                   2023.06.02    h2873b5e_0                    conda-forge/linux-64/re2-2023.06.02-h2873b5e_0.conda                  
  readline              8.2           h8228510_1                    conda-forge/linux-64/readline-8.2-h8228510_1.conda                    
  regex                 2023.10.3     py310h2372a71_0               conda-forge/linux-64/regex-2023.10.3-py310h2372a71_0.conda            
  requests              2.31.0        pyhd8ed1ab_0                  conda-forge/noarch/requests-2.31.0-pyhd8ed1ab_0.conda                 
  s2n                   1.3.56        h06160fa_0                    conda-forge/linux-64/s2n-1.3.56-h06160fa_0.conda                      
  sacremoses            0.0.53        pyhd8ed1ab_0                  conda-forge/noarch/sacremoses-0.0.53-pyhd8ed1ab_0.tar.bz2             
  safetensors           0.3.3         py310hcb5633a_1               conda-forge/linux-64/safetensors-0.3.3-py310hcb5633a_1.conda          
  sentencepiece         0.1.99        hff52083_5                    conda-forge/linux-64/sentencepiece-0.1.99-hff52083_5.conda            
  sentencepiece-python  0.1.99        py310ha7b5816_5               conda-forge/linux-64/sentencepiece-python-0.1.99-py310ha7b5816_5.conda
  sentencepiece-spm     0.1.99        h866249d_5                    conda-forge/linux-64/sentencepiece-spm-0.1.99-h866249d_5.conda        
  setuptools            68.2.2        pyhd8ed1ab_0                  conda-forge/noarch/setuptools-68.2.2-pyhd8ed1ab_0.conda               
  six                   1.16.0        pyh6c4a22f_0                  conda-forge/noarch/six-1.16.0-pyh6c4a22f_0.tar.bz2                    
  snappy                1.1.10        h9fff704_0                    conda-forge/linux-64/snappy-1.1.10-h9fff704_0.conda                   
  sympy                 1.12          pypyh9d50eac_103              conda-forge/noarch/sympy-1.12-pypyh9d50eac_103.conda                  
  tbb                   2021.10.0     h00ab1b0_2                    conda-forge/linux-64/tbb-2021.10.0-h00ab1b0_2.conda                   
  tk                    8.6.13        noxft_h4845f30_101            conda-forge/linux-64/tk-8.6.13-noxft_h4845f30_101.conda               
  tokenizers            0.14.1        py310h320607d_2               conda-forge/linux-64/tokenizers-0.14.1-py310h320607d_2.conda          
  torchtriton           2.0.0         py310                         pytorch/linux-64/torchtriton-2.0.0-py310.tar.bz2                      
  tqdm                  4.66.1        pyhd8ed1ab_0                  conda-forge/noarch/tqdm-4.66.1-pyhd8ed1ab_0.conda                     
  transformers          4.35.0        pyhd8ed1ab_0                  conda-forge/noarch/transformers-4.35.0-pyhd8ed1ab_0.conda             
  typing-extensions     4.8.0         hd8ed1ab_0                    conda-forge/noarch/typing-extensions-4.8.0-hd8ed1ab_0.conda           
  typing_extensions     4.8.0         pyha770c72_0                  conda-forge/noarch/typing_extensions-4.8.0-pyha770c72_0.conda         
  tzdata                2023c         h71feb2d_0                    conda-forge/noarch/tzdata-2023c-h71feb2d_0.conda                      
  ucx                   1.15.0        hae80064_1                    conda-forge/linux-64/ucx-1.15.0-hae80064_1.conda                      
  urllib3               2.1.0         pyhd8ed1ab_0                  conda-forge/noarch/urllib3-2.1.0-pyhd8ed1ab_0.conda                   
  wheel                 0.42.0        pyhd8ed1ab_0                  conda-forge/noarch/wheel-0.42.0-pyhd8ed1ab_0.conda                    
  xxhash                0.8.2         hd590300_0                    conda-forge/linux-64/xxhash-0.8.2-hd590300_0.conda                    
  xz                    5.2.6         h166bdaf_0                    conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2                      
  yaml                  0.2.5         h7f98852_2                    conda-forge/linux-64/yaml-0.2.5-h7f98852_2.tar.bz2                    
  yarl                  1.9.3         py310h2372a71_0               conda-forge/linux-64/yarl-1.9.3-py310h2372a71_0.conda                 
  zipp                  3.17.0        pyhd8ed1ab_0                  conda-forge/noarch/zipp-3.17.0-pyhd8ed1ab_0.conda                     
  zstd                  1.5.5         hfc55251_0                    conda-forge/linux-64/zstd-1.5.5-hfc55251_0.conda
findmyway commented 8 months ago

加环境变量 CUDA_LAUNCH_BLOCKING=1 再试下看看具体的报错信息

jrd77 commented 8 months ago

加环境变量 CUDA_LAUNCH_BLOCKING=1 再试下看看具体的报错信息

ss

findmyway commented 8 months ago

It seems your cuda runtime is not correctly installed.

FYI: https://stackoverflow.com/a/77651517

wells-Qiang-Chen commented 7 months ago

请问你解决了吗，我也是A100 cuda11.8遇到这个问题，加环境变量 CUDA_LAUNCH_BLOCKING=1 后，还是不行 ![Uploading iShot_2024-01-24_08.23.03.png…]()

jrd77 commented 7 months ago

没解决,后续我直接使用huggingFace里面的GPTQ量化的模型,就没问题了,猜测可能是官方使用awq量化,我的显卡支持有问题 'TheBloke/Yi-34B-Chat-GPTQ', 'https://hf-mirror.com/TheBloke/SUS-Chat-34B-GPTQ' @wells-Qiang-Chen