The bug
Initially, I ran the model locally on the computer using models.transformer, took the model from HuggingFace, React worked perfectly.
Next, I tried to write my own custom remote-end-point class (since the same model that I ran locally was raised on my server using openllm) to work with the React structure. And I had a problem that the same model with a hagging face began to constantly go into a loop after the first thought. That is, she began to repeat the structure of Thought-Act-Observation-Thought-Act-Observation, in the role of an assistant.
Next, I tried to use your OpenAi class to work with React and found that the same thing happened. I haven’t tested your other models that access the Remote class, but I suspect that the same thing will happen.
I tried to figure out what was wrong in the Remote class, but I couldn’t. Can you tell me what to do with this?
mkaskov
commented
11 months ago
The bug Initially, I ran the model locally on the computer using models.transformer, took the model from HuggingFace, React worked perfectly.
Next, I tried to write my own custom remote-end-point class (since the same model that I ran locally was raised on my server using openllm) to work with the React structure. And I had a problem that the same model with a hagging face began to constantly go into a loop after the first thought. That is, she began to repeat the structure of Thought-Act-Observation-Thought-Act-Observation, in the role of an assistant.
Next, I tried to use your OpenAi class to work with React and found that the same thing happened. I haven’t tested your other models that access the Remote class, but I suspect that the same thing will happen.
I tried to figure out what was wrong in the Remote class, but I couldn’t. Can you tell me what to do with this?
Below I am attaching a screenshot of what happens when I use your models.OpenAI class and the React structure from the example: https://github.com/guidance-ai/guidance/blob/main/notebooks/chat_with_search.ipynb