search

yGuy / chatgpt-mattermost-bot

A very simple implementation of a service for a mattermost bot that uses ChatGPT in the backend.
MIT License
145 stars 50 forks source link

Bot hallucinates "python" plugin #49

Closed yGuy closed 1 year ago

yGuy commented 1 year ago

This is more like a known issue issue rather than a bug report, but we might consider implementing workarounds for this problem in the ChatGPT LLM in the future.

If you ask ChatGPT mathematical questions, it will hallucinate the existence of a "python" plugin and call that plugin to compute the results. However there is no "python" plugin available. I guess that plugin is available at OpenAI and the LLM was finetuned to make use of that plugin. However we don't have an implementation of that plugin and thus the bot's plugin mechanism detects an error and will tell you:

"Sorry, but it seems there was an error when using the plugin python".

Keeping this around for other people to find if they run into this. If you came across similar messages with non-existent plugins, please report them, here and possibly the prompt that caused it to appear.

yGuy commented 1 year ago

Turns out, whenever you mention "functions" in your prompt, the bot will hallucinate functions, even when there is no need for them - it used the "python" plugin to write python code that used the google translate web service to translate a text, when indeed it can just translate the text without a plugin. When I told it that there was no python plugin, it hallucinated the "translate-plugin"....