search

the-crypt-keeper / LLooM

Experimental LLM Inference UX to aid in creative writing
MIT License
91 stars 8 forks source link

Add the ablity to continue generation. #7

Open cuyler72 opened 4 months ago

cuyler72 commented 4 months ago

Currently if you are not satisfied with the provided options the only way to continue generation is to increase the beam limit and regenerate, It would be nice to be able to continue generation for another cycle.

the-crypt-keeper commented 4 months ago

@cuyler72 This is actually a really interesting thought. The original implementation of the search was depth-first (once a beam is identified we rode its splits all the way until the end, then backend up) but I've since flipped it to breadth-first (for each beam we step one token at a time and keep lots of beams running at once). I originally did this to support parallelism but it turned out to have some other nice properties like being able to support max-suggestions coherently. Now that you mention it, BFS also enables a simple "Step Next Tokens" button that can either supplement or replace the depth and breadth controls. I will look into this one!