search

StampyAI / stampy-chat

Conversational chatbot to answer questions about AI Safety & Alignment based on information retrieved from the Alignment Research Dataset
https://chat.stampy.ai
MIT License
13 stars 6 forks source link

Refresh after full generation to inject content and re-render #76

Closed markovial closed 1 year ago

markovial commented 1 year ago

The current chat generates token by token, but if we are injecting glossary and also rendering equations and stuff, then we probably want to do that once the entire text has been rendered.

So let the thing generate as normal now, then do like a white flash at the end which puts everything through a parser which renders/injects everything that needs to be injected and then re-displays the entire text.

FraserLee commented 1 year ago

As the person who's written the front-end, it's actually not a problem whatsoever to process stuff as we receive it. It gives a far better user experience than waiting (doubly true if we ever make that swap to gpt-4), and even just the one-step-beyond-trivial implementation we've got right now renders absolutely instantly for me. Should that ever not be the case, there's a lot of low hanging fruit optimizations that I suspect could bring rendering performance up by at least an order of magnitude.

It's true - there exist some types of text processing that would need global information, but everything on the roadmap can totally make due either completely chuck by chunk, or with some very minor look-ahead buffer. Even if we add some insane feature down the road, we could reprocess just for that and keep the good UX for everything else.

If you're not satisfied with this answer, I would love to talk through the code model with anyone interested \:)