Closed lawrenceztang closed 2 months ago
I tried phi-2 and llama-3-instruct and they both only generate 16 tokens. How can I change this?
This was confusing to me, but in order to increase the number of tokens to 100 you have to format the request like this:
{ "input": { "prompt": "Hello World", "sampling_params": { "max_tokens": 100 } } }
I tried phi-2 and llama-3-instruct and they both only generate 16 tokens. How can I change this?