search

turboderp / exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs
MIT License
3.19k stars 234 forks source link

Error loading turboderp/Llama-3-70B-Instruct-exl2 #420

Closed pfan94 closed 2 months ago

pfan94 commented 2 months ago 
llm-service-04-1  |   File "/usr/local/lib/python3.10/dist-packages/exllamav2-0.0.18-py3.10-linux-x86_64.egg/exllamav2/config.py", line 226, in prepare
llm-service-04-1  |     f = STFile.open(st_file, fast = self.fasttensors, keymap = self.arch.keymap)
llm-service-04-1  |   File "/usr/local/lib/python3.10/dist-packages/exllamav2-0.0.18-py3.10-linux-x86_64.egg/exllamav2/fasttensors.py", line 114, in open
llm-service-04-1  |     return STFile(filename, fast, keymap)
llm-service-04-1  |   File "/usr/local/lib/python3.10/dist-packages/exllamav2-0.0.18-py3.10-linux-x86_64.egg/exllamav2/fasttensors.py", line 67, in __init__
llm-service-04-1  |     self.read_dict()
llm-service-04-1  |   File "/usr/local/lib/python3.10/dist-packages/exllamav2-0.0.18-py3.10-linux-x86_64.egg/exllamav2/fasttensors.py", line 126, in read_dict
llm-service-04-1  |     header_size = np.fromfile(fp, dtype = np.int64, count = 1).item()
llm-service-04-1  | ValueError: can only convert an array of size 1 to a Python scalar

I downloaded turboderp/Llama-3-70B-Instruct-exl2 and use 6.0bpw branch, get this eror while loading

turboderp commented 2 months ago

Can you check that you've downloaded the files correctly? E.g.

(venv) [bb@bbc 6.0bpw]$ sha256sum *.safetensors
2b18d0f2bcddbe5576e135029c4555f96b3470d1a730e86a241db84154382d0a  output-00001-of-00007.safetensors
f8afa62f51e0bc1aad200cb6930ff04549a90cd91fefaf7c5d1ba19a753fc24b  output-00002-of-00007.safetensors
5079e03004126601b9f96defc98afed0cd6f0e9f47ad7f31716d0538fa772d25  output-00003-of-00007.safetensors
04cb7ba9a8711311a548c492bdfd9ce967f32381c6b79727cd5deb23e725249c  output-00004-of-00007.safetensors
05ce582daeb11628d698e6136b4f3dde885ec0b582fc363e10bcf17e2be2a684  output-00005-of-00007.safetensors
74a55bf96eb07055f5a6c7b59e30191b6ae0edc1764f41167641ecc9115867c2  output-00006-of-00007.safetensors
aba245c40fe949a33aa54eb9d5d4798109e295944ddeefaf3f93572792eb0ec5  output-00007-of-00007.safetensors
pfan94 commented 2 months ago

Can you check that you've downloaded the files correctly? E.g.

(venv) [bb@bbc 6.0bpw]$ sha256sum *.safetensors
2b18d0f2bcddbe5576e135029c4555f96b3470d1a730e86a241db84154382d0a  output-00001-of-00007.safetensors
f8afa62f51e0bc1aad200cb6930ff04549a90cd91fefaf7c5d1ba19a753fc24b  output-00002-of-00007.safetensors
5079e03004126601b9f96defc98afed0cd6f0e9f47ad7f31716d0538fa772d25  output-00003-of-00007.safetensors
04cb7ba9a8711311a548c492bdfd9ce967f32381c6b79727cd5deb23e725249c  output-00004-of-00007.safetensors
05ce582daeb11628d698e6136b4f3dde885ec0b582fc363e10bcf17e2be2a684  output-00005-of-00007.safetensors
74a55bf96eb07055f5a6c7b59e30191b6ae0edc1764f41167641ecc9115867c2  output-00006-of-00007.safetensors
aba245c40fe949a33aa54eb9d5d4798109e295944ddeefaf3f93572792eb0ec5  output-00007-of-00007.safetensors

thanks for fast response. Here is my output, some files are incorrect downloaded. I will try again :)

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  output-00001-of-00007.safetensors
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  output-00002-of-00007.safetensors
5079e03004126601b9f96defc98afed0cd6f0e9f47ad7f31716d0538fa772d25  output-00003-of-00007.safetensors
04cb7ba9a8711311a548c492bdfd9ce967f32381c6b79727cd5deb23e725249c  output-00004-of-00007.safetensors
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  output-00005-of-00007.safetensors
579ae9ff616aa7766c61622ab9f4911313a6fcdbb2d7689b05f2af180411fe55  output-00006-of-00007.safetensors
aba245c40fe949a33aa54eb9d5d4798109e295944ddeefaf3f93572792eb0ec5  output-00007-of-00007.safetensors