search

EleutherAI / lm-evaluation-harness

A framework for few-shot evaluation of language models.
https://www.eleuther.ai
MIT License
6.61k stars 1.74k forks source link

OpenaiCompletionsLM invokes the completions API with max_tokens set to 0 #1903

Open chimezie opened 4 months ago

chimezie commented 4 months ago

As per the title, the completions API is invoked with max_tokens = 0, which, if properly interpreted by the server, will cause it not to generate anything (according to the API documentation, that value defaults to 16 and there is no indication that a value of 0 has any special meaning).

It seems the value of self._max_gen_toks (via the max_gen_toks property) should have been used instead.

1m0d commented 4 months ago

I have the same issue when using the local completions model (vLLM with OpenAI server) I am getting the following error: openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': 'max_tokens must be at least 1, got 0.', 'type': 'BadRequestError', 'param': None, 'code': 400}

Additionally, the logprobs parameter is set to 10 even though the maximum value as per the OpenAI documentation is 5. https://github.com/EleutherAI/lm-evaluation-harness/blob/070d31df4edad2d8263b64b3ace7ec0cbaeb7ae8/lm_eval/models/openai_completions.py#L215 https://platform.openai.com/docs/api-reference/completions/create#completions-create-logprobs