[GEMMA] `from_hugging_face` not setting `share_embedding_table` to True leading to incapacity to load Gemma

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

https://nvidia.github.io/TensorRT-LLM

Apache License 2.0

7.34k stars 794 forks source link

[GEMMA] `from_hugging_face` not setting `share_embedding_table` to True leading to incapacity to load Gemma #1799

Closed mfuntowicz closed 3 days ago

mfuntowicz commented 1 week ago

Gemma requires shared_embedding_table to be True but coming from from_hugging_face doesn't set this parameter.

nv-guomingz commented 1 week ago

@mfuntowicz Thanks for your contribution to TRT-LLM, we'll take a look this change firstly.

MartinMarciniszyn commented 1 week ago

Opening an internal MR for that.

nv-guomingz commented 1 week ago

Hi @mfuntowicz , thanks for your contribution, we've merged your changes into our internal code base and will upstream to main branch next week

kaiyux commented 3 days ago

Hi @mfuntowicz , the main branch has been updated with your changes integrated, thanks a lot for your contribution.

I'm closing this PR, please let us know if there are any questions. Thanks.