search

PrivateCoffee / matrix-gptbot

GPT Chatbot for Matrix
Other
8 stars 3 forks source link

"500 Internal Server Error" returned by Gemini when sending pictures in encrypted rooms #11

Open dillfrescott opened 1 month ago

dillfrescott commented 1 month ago

How complicated is the whole issue with the bot not being able to see encrypted images with panta?

Is that an issue on pantas side? or is that something you think you could fix potentially?

dillfrescott commented 1 month ago

Because right now if i send any image in the encrypted chat with the bot it completely breaks it.

dillfrescott commented 1 month ago

which is a big problem, could the bot just ignore the image maybe?

kumitterer commented 1 month ago

As I see it, it is an issue with Pantalaimon that gets confused with unencrypted images, tries to decrypt them anyway and fails, but I'm not entirely sure about that. I'll have to investigate that further.

That said, it shouldn't crash the bot completely, it should just disregard the image. Do you see anything in the logs that might be relevant?

dillfrescott commented 1 month ago

yes

[2024-05-16 07:11:56:702383] - generate_chat_response - [INFO] Generated response: [GoogleGenerativeAI Error]: Error fetching from google -> [500 Internal Server Error] An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

im not sure what its sending, but it works fine in unencrypted rooms and doesnt throw the error 500, so im guessing its not on googles side of things.

dillfrescott commented 1 month ago

it is still trying to send the image data to the model it seems even though the room is encrypted

dillfrescott commented 1 month ago

To clarify, it identifies images perfectly fine using the gemini 1.5 flash model in unencrypted rooms

kumitterer commented 1 month ago

Hmm. I've never tested with Gemini, but if I had to guess, Google's API behaves differently from the ones I've tested so far, and returns a server error (which could really be more verbose) if it is passed a file it can't process, as opposed to OpenAI or ollama that just quietly ignore the file.

What's almost more confusing – in my experience, errors processing images seemed to occur in unencrypted rooms, not in encrypted ones.

Gonna have to test that. :thinking:

dillfrescott commented 1 month ago

Indeed it is weird. My openai to gemini proxy im running on vercel is set to send every request to the flash gemini model, which supports image vision.

I modified this line to just say "gemini" instead of "vision" and that enabled the use of gemini to see and describe images. Now im just trying to figure out why its throwing the error 500 with pantalaimon hooked up.

Ideally id love for it to respond to images in encrypted rooms but that might not be possible so ill settle with it just not breaking completely when i send an image in an encrypted room.

Is there a way you can add an explicit check if a room is encrypted and just never even put the base64 of the image or anything in the response to the model?

I would greatly appreciate that!

dillfrescott commented 1 month ago

Also to add, it responds with text just fine in encrypted rooms. We can chat back and forth, but the second i send an image it sends blank responses, until i run !gptbot ignoreolder.

dillfrescott commented 1 month ago

Screenshot_20240516_034821 as you can see it works spectacularly in unencrypted rooms. leading me to think the error 500 might be the result of a malformed send response to the model.

dillfrescott commented 1 month ago

aaa and an example of how it behaves in an encrypted room

dillfrescott commented 1 month ago

also ive tried the panta device verification and it refuses to actually send the device verification prompt to the other client. So i have settled with it having the little red shield indicating it is an unverified session.

kumitterer commented 1 month ago

Thanks for all of those details – I'm trying to reproduce the issue. However, for me, when I send an image file and it gets scrambled by Pantalaimon, I get an error like this one in the logs:

[2024-05-17 10:59:42:001844] - download_file - [ERROR] Error downloading file: Internal server error

And after that, the file should be ignored completely.

https://github.com/PrivateCoffee/matrix-gptbot/blob/bc06f8939a8611cce669de998eb9451f8e7152b7/src/gptbot/classes/bot.py#L1362-L1378

https://github.com/PrivateCoffee/matrix-gptbot/blob/bc06f8939a8611cce669de998eb9451f8e7152b7/src/gptbot/classes/bot.py#L1263-L1271

If you don't see that error, something else is going on. Could you maybe share your configuration file (sensitive data redacted, of course) and the OpenAI to Gemini proxy you are using? Then I can try to reproduce that.

dillfrescott commented 1 month ago

I apologize! I removed panta and am just using the bot in unencrypted rooms. Here is the proxy if you'd still like to take a look at it!

kumitterer commented 1 month ago

Thanks for that - I'm definitely still going to look into this issue, it's really rather annoying.

FYI, I have just pushed a new tag that allows enabling vision on third-party models, so you don't have to modify the code anymore:

https://github.com/PrivateCoffee/matrix-gptbot/blob/9178ab23ac8ce90eea06fdd0597612c76f7323a9/config.dist.ini#L124-L131

dillfrescott commented 1 month ago

Oh thats neat! I appreciate that!