djparente
commented
1 month ago I've been working on this same problem with a Llama-3.1 exl2 model against rc054. I think part of the problem is that the model.safetensors.index.json file is pointing at files that do not exist. For example, in the turboderp Llama 3.1 8B repository model.safetensors.index.json is calling for "model-00004-of-00004.safetensors" which doesn't exist in that exl2 repo (there is a single model shard called output.safetensors).
This problem also exists in other Llama 3.1 exl2 repositories (e.g., two 70B repositories LoneStriker/Meta-Llama-3.1-70B-Instruct-4.0bpw-h6-exl2 and turboderp/Llama-3.1-70B-Instruct-exl2)
I had claude write me a script to introspect into the safetensors file and generate a corrected model.safetensors.index.json file (see below). This gets rid of the error that it cannot find any model weights but there are further problems.:
First, it errors that "ValueError: torch.bfloat16 is not supported for quantization method exl2. Supported dtypes: [torch.float16]"
Suppose you try to fix this by explicitly setting the --dtype to float16 (I'm not sure if this is legitimate or not), then you run into a further error (also below). I looked at the code for a fair bit and tried to debug, but I think this perhaps is because the exl2 model is quantizing lm_head whereas the loader is not expecting this to be quantized?
So, possible actionable fixes for aphrodite:
- Implement bfloat16 support for exl2?
- Update the loader to support a quantized header (or otherwise support layers for lm_head in the exl2 safetensors files).
It is possible I do not understand the issues in enough detail to have pinned this down. Any other information you have on this topic would be much appreciated.
Much appreciate your wonderful work on this project.
[rank0]: File "/home/dan/.venvs/aphrodite-rc054-dev/bin/aphrodite", line 8, in <module>
[rank0]: sys.exit(main())
[rank0]: ^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/endpoints/cli.py", line 25, in main
[rank0]: args.func(args)
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 542, in run_server
[rank0]: engine = AsyncAphrodite.from_engine_args(engine_args)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 391, in from_engine_args
[rank0]: engine = cls(
[rank0]: ^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 346, in __init__
[rank0]: self.engine = self._init_engine(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/engine/async_aphrodite.py", line 466, in _init_engine
[rank0]: return engine_class(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/engine/aphrodite_engine.py", line 203, in __init__
[rank0]: self.model_executor = executor_class(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/executor/multiproc_gpu_executor.py", line 135, in __init__
[rank0]: super().__init__(*args, **kwargs)
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/executor/distributed_gpu_executor.py", line 24, in __init__
[rank0]: super().__init__(*args, **kwargs)
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/executor/executor_base.py", line 42, in __init__
[rank0]: self._init_executor()
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/executor/multiproc_gpu_executor.py", line 62, in _init_executor
[rank0]: self._run_workers("load_model",
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/executor/multiproc_gpu_executor.py", line 113, in _run_workers
[rank0]: driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/task_handler/worker.py", line 124, in load_model
[rank0]: self.model_runner.load_model()
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/task_handler/model_runner.py", line 138, in load_model
[rank0]: self.model = get_model(
[rank0]: ^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/modeling/model_loader/__init__.py", line 21, in get_model
[rank0]: return loader.load_model(model_config=model_config,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/modeling/model_loader/loader.py", line 241, in load_model
[rank0]: model.load_weights(
[rank0]: File "/home/dan/aphrodite/rc054/aphrodite-engine/aphrodite/modeling/models/llama.py", line 430, in load_weights
[rank0]: param = params_dict[name]
[rank0]: ~~~~~~~~~~~^^^^^^
[rank0]: KeyError: 'lm_head.q_groups'Introspection script below to re-write the models.safetensors.index.json file from analysis of the safetensors files. (Not sure if the total size is being calculated correctly).
import json
from safetensors import safe_open
import os
def get_weights_in_file(filename):
weights = {}
with safe_open(filename, framework="pt", device="cpu") as f:
for key in f.keys():
tensor = f.get_tensor(key)
weights[key] = {
"shape": list(tensor.shape),
"dtype": str(tensor.dtype)
}
return weights
def create_model_index(directory):
weight_map = {}
metadata = {"total_size": 0}
for filename in os.listdir(directory):
if filename.endswith('.safetensors'):
full_path = os.path.join(directory, filename)
file_size = os.path.getsize(full_path)
metadata["total_size"] += file_size
weights = get_weights_in_file(full_path)
for weight_name in weights:
weight_map[weight_name] = filename
index = {
"metadata": metadata,
"weight_map": weight_map
}
return index
def write_model_index(directory, output_file="model.safetensors.index.json"):
index = create_model_index(directory)
output_path = os.path.join(directory, output_file)
with open(output_path, 'w') as f:
json.dump(index, f, indent=2)
print(f"Created {output_file} with {len(index['weight_map'])} weights mapped.")
if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print("Usage: python script_name.py <directory_path>")
sys.exit(1)
directory = sys.argv[1]
write_model_index(directory)
AlpinDale
Your current environment
🐛 Describe the bug
When trying to run any exl2 model, the following error occurs, even if you manually download the model and specify the path for it: