Open gserapio opened 7 months ago
CC: @stephenfitz
Thank you for opening an issue! Is there any way you could run this in a debugger and tell me what is passed to self.tokenizer.encode
? I suspect a Numpy
array, but I would like to be sure.
Thanks for all of your work on this resource! Yes, it looks like response
is a numpy
array when it's passed to self.tokenizer.encode
. Here's the debugger output:
> ~/opt/anaconda3/envs/myenv/lib/python3.10/site-packages/outlines/models/openai.py(220)generate_choice()->None
-> encoded_response = self.tokenizer.encode(response)
(Pdb) l
215 prompt, system_prompt, self.client, config
216 )
217 self.prompt_tokens += prompt_tokens
218 self.completion_tokens += completion_tokens
219
220 -> encoded_response = self.tokenizer.encode(response)
221
222 if encoded_response in encoded_choices_left:
223 decoded.append(response)
224 break
225 else:
(Pdb) p response
array(['Blue', 'Red', 'Blue'], dtype='<U4')
(Pdb) whatis response
<class 'numpy.ndarray'>
Describe the issue as clearly as possible:
One of the nice features of using Outlines with HF models is the ability to vectorize calls to models rather than making calls serially.
Passing a list prompts to an OpenAI model via
outlines.generate.choice()
results in aTypeError
. The responses from OpenAI API are converted into anumpy
array before being encoded bytiktoken
, which throws the error. Perhaps it has something to do with that?The issue doesn't occur when a single prompt is passed to OpenAI as a string.
My reprex uses the latest version of Outlines, 0.0.36, but I've also encountered this issue in previous versions.
Steps/code to reproduce the bug:
Expected result:
Error message:
Outlines/Python version information:
Version information
Context for the issue:
The current workaround for this issue is to serially send prompts to your model generator in a for loop, but it would be nice to get vectorized calls to work so that OpenAI and transformers model objects can be used interchangeably in bulk inference pipelines.