Default value of max_tokens is set to 16, which does not generate good response for most of the scenario. Setting it to something higher value similar to openai can be helpful.
This is mostly my experience, obviously there might be other scenario where folks will prefer smaller values. Still 16 token does not even generate complete a sentence or two sometime.
mattf
commented
5 months ago
Default value of
max_tokensis set to 16, which does not generate good response for most of the scenario. Setting it to something higher value similar to openai can be helpful.This is mostly my experience, obviously there might be other scenario where folks will prefer smaller values. Still 16 token does not even generate complete a sentence or two sometime.
Will love to hear your opinion on this.