rodrigo-f-nogueira
commented
10 months ago Hi @willianfco, Yes, we've update the API with a new model, hence the faster inference and different behaviour. From now on, we will introduce a versioning system visible to the user so it does not cause any surprises (sorry about that).
We will also have an updated version with better answers in a month or so.
Hello team, hope you are all well!
Issue Description:
When I end my prompts with "Resposta: " to ensure the generation of concise answers without leading tabs or linebreaks, the API has started returning a blank space (" ") as the response. This behavior is new and wasn't observed in earlier usage.
With the same prompt but "Resposta:" (without the blank space at the end) it works, but the quality of the inference noticeably deteriorates compared to previous tests (with identical prompts and parameters).
We used the following generation parameters:
model.generate(query, chat_mode=False, do_sample=True, max_tokens=500, temperature=0.1, top_p=0.15)Another point I've noticed is that the inference process seems to be faster now. Could it be possible that there have been recent changes to the base model or its configuration on the API?
Your insights or suggestions on resolving this issue would be greatly appreciated.
Thank you for your assistance.