Great Work!
Looking at the code my understanding is that even though LEVEL/GUESS_SIZE is configurable, it is fixed for the entire inference for a given input prompt. I was wondering if you have looked into dynamically changing this based on the max_hits value? or increasing it in some sort of scheduled manner based on the number of output tokens generated?
Great Work! Looking at the code my understanding is that even though
LEVEL/GUESS_SIZEis configurable, it is fixed for the entire inference for a given input prompt. I was wondering if you have looked into dynamically changing this based on themax_hitsvalue? or increasing it in some sort of scheduled manner based on the number of output tokens generated?