Prevent duplicate `torch_dtype` kwargs

microsoft / LLMLingua

To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.

https://llmlingua.com/

MIT License

4.18k stars 222 forks source link

Prevent duplicate `torch_dtype` kwargs #115

Closed yasyf closed 3 months ago

yasyf commented 3 months ago

Throws an error currently if you specify a torch_dtype in the model_config.

TypeError: transformers.models.auto.auto_factory._BaseAutoModelClass.from_pretrained() got multiple values for keyword argument 'torch_dtype'

cc: @iofu728, @QianhuiWu, @XufangLuo, @mydmdm

iofu728 commented 3 months ago

Thank you for @yasyf's help. It's merged now.