Fix token limits - Githubissues

cboard-org / cboard-ai-engine

This engine powers the Cboard AI builder, designed to generate content suggestions for communication boards and create new pictograms as necessary. With a simple prompt, it will generate a list of suggestions that can be used to create an AAC board

https://cboard.io

GNU General Public License v3.0

1 stars 1 forks source link

Fix token limits #26

Closed hectoritr closed 8 months ago

Da-PC commented 8 months ago

I´ve run multiple runs and this the data I´ve gathered. My assumptions are:

Max Pictos per request 50.

Results: If we provide a high number of token the Chat get dilusional and add repeated words or garbage, so we need to constrain the token limit. This also helps with security and jailbreaking. The growth of token is 4.11x+76.7. I´ll be using 4x+80 to accomodate word lenght variation based on the topic of the prompt.

Da-PC commented 8 months ago

Final comment, with higher number is still a bit delusional and return more suggestions than requested.

tomivm commented 8 months ago

Good! Do you think that #22 is caused by this token limit? If we could return the data before the limit would be better than returning an empty array. Also if this is not possible would be better to throw an Error

Da-PC commented 8 months ago

Yes I think this will be solved. But still the Error handling will be good practice.