Closed ZQ-Dev8 closed 7 months ago
For additional context - I retrained the model, this time with flash attention, and the error remains the same:
RuntimeError: weight gpt_neox.layers.0.attention.rotary_emb.inv_freq does not exist
rank=0
2024-01-20T14:17:26.514770Z ERROR text_generation_launcher: Shard 0 failed to start
2024-01-20T14:17:26.514815Z INFO text_generation_launcher: Shutting down shards
Error: ShardCannotStart
Failed to run text-generation-launcher.
@Narsil I think this is probably the same as https://github.com/huggingface/text-generation-inference/issues/790 for Llama which you fixed by removing loading of position embeddings from weights in https://github.com/huggingface/text-generation-inference/pull/793. This fix seems simple enough but it looks like there was some concern about it breaking previous versions of the model / integration tests (but maybe everything was fine)
I think this probably broke when position embeddings were removed from the GPTNeoX weights (link to line diff from @ArthurZucker), which looks to have been released in transformers 4.35 so if you don't need Flash Attention, you might be able to downgrade transformers and re-export the model to get things working until fixed
TYVM for the response. I can see the PR is held up - I'll attempt the downgrade/re-export you mentioned in the meantime.
System Info
docker run --rm --entrypoint /bin/bash -itd --name "traclm-v1-3b-instruct" -v "path/to/folder":/data --gpus '"device=3"' -p 172.20.158.30:8082:80 ghcr.io/huggingface/text-generation-inference:latest
togethercomputer/RedPajama-INCITE-Base-3B-v1
withteknium/GPT4-LLM-Cleaned
, created using the HF Trainer without flash attention (could this be the problem?)-Deployment specificities: N/A, using latest TGI image as of creating this issue.
Information
Tasks
Reproduction
N/A, I am attempting to load the model via standard usage of TGI. The problem is related to the model, which uses the GPTNeoX architecture (pasted below). Note the model was not trained with flash attention, though the provided stacktrace seems to imply it needs to be (e.g. multiple instances of
FlashNeoXAttention()
)Expected behavior
I would expect the TGI server to start as with any other supported model. However, for this model I get the following error: