Open phishmaster opened 2 months ago
I am having the same issue
Problem with max_tokens
less than n_ctx
. I think we need to add an assert to ensure context bigger than generated text size.
Problem with
max_tokens
less thann_ctx
. I think we need to add an assert to ensure context bigger than generated text size.
max_tokens
is 1024 while n_ctx
is 200 in the provided example though. Do you mean that n_ctx should be greater than the actual context window + the output tokens?
Problem with
max_tokens
less thann_ctx
. I think we need to add an assert to ensure context bigger than generated text size.
max_tokens
is 1024 whilen_ctx
is 200 in the provided example though. Do you mean that n_ctx should be greater than the actual context window + the output tokens?
Yes, greater, not less. My mistake.
Did you try increasing the max output tokens (in this test, try setting it to eg 20k or so, just to be sure). Does this solve the issue? As long as the input does not exceed the context (which should error out) I don't think the context is involved.
I think this bug happens when the input is smaller than n_ctx
, but the input + output is greater than n_ctx
.
Installed from conda environment with pip Version: '0.2.69' The code is as follow:
Error message