search

FasterDecoding / Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
https://sites.google.com/view/medusa-llm
Apache License 2.0
2.28k stars 155 forks source link

Add a simple gradio interface, make life easier #23

Closed Mrw33554432 closed 1 year ago

Mrw33554432 commented 1 year ago

A simple interface with gradio, not much to talk about. The tricky part is, the model always regard itself as the user after a few chats, but that's not a super big issue (not related to the interface I think).

ctlllll commented 1 year ago

Thanks for the contribution! The code LGTM. Could you please also add a few arguments like model name and temperature, as in the command line CLI?

Mrw33554432 commented 1 year ago

Thanks for the contribution! The code LGTM. Could you please also add a few arguments like model name and temperature, as in the command line CLI?

Added a model loader page and some options like temperature/max step/no history. But I faced some issue while trying to add a logo. I tested the code with 7b model, and it works as expected. However, the quant options are not tested (windows system).