search

PygmalionAI / gradio-ui

Prototype UI for chatting with the Pygmalion models.
GNU Affero General Public License v3.0
225 stars 66 forks source link

Experimental version of the 6B might be broken after the latest Kobold United update #28

Closed 0x000011b closed 1 year ago

0x000011b commented 1 year ago

As it turns out, apparently there was a bug in Kobold that made it ignore the model version we were passing in, so anyone using KAI as an inference back-end (that is: literally everyone using the notebook on Colab) wasn't actually using the experimental version, even if they selected it.

Henk has fixed this last night, but by default Kobold bans the generation of EOS tokens, which is troublesome for us because the SFT models generate an EOS after their response to stop the generation, but because of the token ban on Kobold they'll just start generating trash instead.

Need to confirm whether this is actually happening now, and if it is, we need to un-ban the EOS token for all SFT models.

0x000011b commented 1 year ago

@TearGosling I assigned you on this one as well in case you can keep up with the dev talk and get to this quicker than me. Kinda important because there's ~20k daily clones of the notebook, and I imagine a non-significant portion of these users are trying the experimental version and so are being affected by the bug now.

TearGosling commented 1 year ago

Definitely broken. Looking into trying to get the EOS token unbanned right now.

TearGosling commented 1 year ago

Update: tests on the model without Kobold show the experimental model is fine when generating from pure HuggingFace. This may be a problem localized to Kobold. Work continues.

0x000011b commented 1 year ago

Worth labeling the experimental version as broken on the notebook until we fix this?

TearGosling commented 1 year ago

Will do.