search

the-crypt-keeper / LLooM

Experimental LLM Inference UX to aid in creative writing
MIT License
74 stars 6 forks source link

Add a suggestion hard limiter #6

Closed the-crypt-keeper closed 1 month ago

the-crypt-keeper commented 1 month ago

In Auto-Depth mode even with the Split Limit the number of suggestions can get out of hand once you get deeper into inference. There should be a way to limit the total number of beams returned so that it doesn't sit just there and run forever.

For these dense cases you probably want to disable Auto-Depth and manually step 6-8 tokens at a time until you get out of the woods.

the-crypt-keeper commented 1 month ago

Maximum Suggestions is available in the latest version. This is not quite a hard limiter for reasons that are fairly complex, but beams will stop splitting after this threshold and in general you should end up within 10%.