search

google / jetstream-pytorch

PyTorch/XLA integration with JetStream (https://github.com/google/JetStream) for LLM inference"
Apache License 2.0
32 stars 14 forks source link

Support converting hf gemma weights #71

Closed lsy323 closed 3 months ago

lsy323 commented 3 months ago

Tested with running convert_checkpoint.py and run_interactive.py

qihqi commented 3 months ago

Can you also add the instructions in your PR to the README.md? thanks!

lsy323 commented 3 months ago

Can you share the real output result with the gemma weights?

There is some issue with the response, it's generating English but it will repeat the sentence. @qihqi will update the attention module for Gemma in the following PR, which should fix the accuracy issue.

Prompt

"I believe the meaning of life is"

The response is:

 to experience.

Here is my reasoning:

* **Life is a journey.** We are all on a journey through time. The experiences we gain through life's journey is to experience the world and all that we experience, both positive and negative experiences, both good and bad experiences, both positive and negative experiences, both joy and suffering. We all go through life to experience and learn from both good and bad and positive and negative and positive and negative experiences, both joy and suffering and pain and joy, both good and bad and good and bad and negative and positive and negative and positive and negative and good and bad and the good and bad and negative and positive and negative and positive and negative and the good and bad and negative and positive and negative experiences, both good and bad and the good and bad and negative and the good and bad. The is the good and bad and the good and bad and the the good and bad and the the are the experiences. Through the joy and negative. The the the journey is the good. The through the the.
FanhaiLu1 commented 3 months ago

Can you share the real output result with the gemma weights?

There is some issue with the response, it's generating English but it will repeat the sentence. @qihqi will update the attention module for Gemma in the following PR, which should fix the accuracy issue.

Prompt

"I believe the meaning of life is"

The response is:

 to experience.

Here is my reasoning:

* **Life is a journey.** We are all on a journey through time. The experiences we gain through life's journey is to experience the world and all that we experience, both positive and negative experiences, both good and bad experiences, both positive and negative experiences, both joy and suffering. We all go through life to experience and learn from both good and bad and positive and negative and positive and negative experiences, both joy and suffering and pain and joy, both good and bad and good and bad and negative and positive and negative and positive and negative and good and bad and the good and bad and negative and positive and negative and positive and negative and the good and bad and negative and positive and negative experiences, both good and bad and the good and bad and negative and the good and bad. The is the good and bad and the good and bad and the the good and bad and the the are the experiences. Through the joy and negative. The the the journey is the good. The through the the.

Great! The result is reasonable.

lsy323 commented 3 months ago

Can you also add the instructions in your PR to the README.md? thanks!

Added to the README.md for how to run Gemma.

lsy323 commented 3 months ago

Updated convert_checkpoint.py to generate weight in safetensor format by default. Loading weights in state_dict is still added in the weight loading logic.

FanhaiLu1 commented 3 months ago

Updated convert_checkpoint.py to generate weight in safetensor format by default. Loading weights in state_dict is still added in the weight loading logic.

Great! All looks good to me now.