search

danikhan632 / guidance_api

An Extension for oobabooga/text-generation-webui
35 stars 7 forks source link

Thank you! #2

Closed DifferentialityDevelopment closed 1 year ago

DifferentialityDevelopment commented 1 year ago

Thank you very much for this, guidance is an amazing tool and this will massively assist in using it alongside all the goodness text-generation-webui brings together.

If you don't mind me asking, how does it integrate guidance with text-generation-webui, I have a rough idea of how guidance works but I'm totally curious how guidance is integrated into the text-generation-webui pipeline.

danikhan632 commented 1 year ago

You enable the extension by settings as an arg like this

CMD_FLAGS = " --chat --model-menu  --model decapoda-research_llama-7b-hf --extensions guidance_api"

currently I have an Open PR to guidance https://github.com/microsoft/guidance/pull/221

then once running you would initalize guidance like this

guidance.llm = guidance.llms.TGWUI("http://127.0.0.1:9555")
DifferentialityDevelopment commented 1 year ago

Awesome! Let's hope they can merge the PR soon :) This will bring so much value to the whole community, guidance is incredibly useful for so so many laborious tasks.

danikhan632 commented 1 year ago

Thank you so much for the feedback looking forward to a merged pr :)