search

PrivateCoffee / matrix-gptbot

GPT Chatbot for Matrix
Other
8 stars 3 forks source link

Request: Video support #15

Closed dillfrescott closed 1 month ago

dillfrescott commented 1 month ago

I know gemini flash has the ability to describe videos. In the api playground it even gives you some videos up to 5 minutes to send it so it can describe them.

Is this something you could add into the bot? Im not sure how the videos need to be prepared, if they need to be broken into frames maybe? but This would be a super cool addition!

kumitterer commented 1 month ago

Oh, that looks interesting. According to https://ai.google.dev/gemini-api/docs/prompting_with_media?lang=python, videos can be passed like any other file, so it shouldn't be too hard to get this working.

dillfrescott commented 1 month ago

Oh perfect! I would love this feature!

kumitterer commented 1 month ago

I just tagged a new v0.3.15 release that might support this feature. :sweat_smile:

I haven't tested it myself, because I haven't set up the proxy yet (and I have started implementing Gemini support directly into the bot, but that's not ready yet). But theoretically, if you send a Matrix video message, or a file that ends in .mp4, it should now be passed to the API in the same way images are. You need to set ForceVideoInput=1 in the config file's OpenAI section to enable this feature.