mgoin
commented
2 months ago @sadrafh I was able to get this model to load with a small fix to the config:
diff --git a/vllm/transformers_utils/configs/falcon.py b/vllm/transformers_utils/configs/falcon.py
index c82cc606..43eb0438 100644
--- a/vllm/transformers_utils/configs/falcon.py
+++ b/vllm/transformers_utils/configs/falcon.py
@@ -69,6 +69,7 @@ class RWConfig(PretrainedConfig):
self.bias = bias
self.parallel_attn = parallel_attn
self.new_decoder_architecture = new_decoder_architecture
+ self.num_ln_in_parallel_attn = None
if self.hidden_size == 8192:
# Hack for falcon-40bHowever it seems to produce gibberish:
>>> from vllm import LLM
>>> model = LLM("amazon/FalconLite2", trust_remote_code=True, quantization="gptq")
>>> model.generate("<|prompter|>What are the main challenges to support a long context for LLM?<|endoftext|><|assistant|>")
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 3.64it/s, est. speed input: 62.00 toks/s, output: 58.35 toks/s]
[RequestOutput(request_id=0, prompt='<|prompter|>What are the main challenges to support a long context for LLM?<|endoftext|><|assistant|>', prompt_token_ids=[65028, 1562, 362, 248, 1316, 4922, 271, 1164, 241, 916, 4436, 312, 31370, 56, 42, 11, 65027], prompt_logprobs=None, outputs=[CompletionOutput(index=0, text=' Ern Solar approachedThomas kost lol igual Alarm creatorvenue交Verified Debor soybean circulatecaster', token_ids=(53224, 15197, 15377, 24174, 33011, 12929, 26976, 41954, 17087, 4255, 20098, 48622, 37688, 52315, 54369, 30414), cumulative_logprob=None, logprobs=None, finish_reason=length, stop_reason=None)], finished=True, metrics=RequestMetrics(arrival_time=1722542626.8144124, last_token_time=1722542626.8144124, first_scheduled_time=1722542626.8371897, first_token_time=1722542626.9011965, time_in_queue=0.022777318954467773, finished_time=1722542627.1092346), lora_request=None)]
sadrafh
I want to use vllm and the model amazon/FalconLite2 which can be found https://huggingface.co/amazon/FalconLite2 for benchmarking throughput and latencies. However, the model is not supported by vllm. What should I do to make this run? Thanks