Add support for a GIF keyboard / ability to easily send GIFs

[matrixbot]

Created by @ matthew:matrix.org.

[uhoreg]

(So I lied. But this one is really my last comment for the night. I promise. Honestly.) Would this be covered by sticker support, if the stickers could be GIFs?

[non-Jedi]

I think this is covered by the sticker api at least. Probably can close this one.

[ara4n]

...except it isn't, as the sticker API isn't a GIF keyboard. I can't search for "arrested development" and get a selection of memes to pick from.

[SimmyD]

As i get friends onto riot mostly the first thing is "how do i send and search gifs"

Is there an opposition to looking at adding something like giphy directly into the UI?

[pixlwave]

Having this same conversation with someone right now. Their words not mine:

Also, it won't be mainstream until it has gifs gifs are everything

🤔

[motey]

My boss comes in my office once a week and wants to have an update on the "gif-thing-status in the new chat software"

[J-tt]

This really feels like something that should be in Element, I've put a bounty on it:

https://www.bountysource.com/issues/28576328-we-need-a-gif-keyboard-toooooo-11-1

[Liam-Lumity]

This really feels like something that should be in Element, I've put a bounty on it:

https://www.bountysource.com/issues/28576328-we-need-a-gif-keyboard-toooooo-11-1

Added to this, no chat app can persist in this world of memes and custom emojis without the functionality to share them easily

[J-tt]

The bounty is now at US$125, after an additional contribution by myself and itsthebatman

[J-tt]

We've reached US$200 which was the goal set by @denim2x so hopefully we get some action soon!

[ShadowJonathan]

Discord uses giphy, while i think this is a bit weird to have jammed into a privacy-oriented and generic messenger, i think at least giving the option of sending GIFs straight from there would be useful. (Discord has a /giphy command that pops up suggested GIFs (which you can click on), and picks the first one when pressing enter)

[tohojo]

Jonathan de Jong @.***> writes:

Discord uses giphy, while i think this is a bit weird to have jammed into a privacy-oriented and generic messenger,

So does the Signal Android app, but IIRC they proxy the requests so it's not possible to correlate them to specific clients. I guess element could do that too (letting the user set the proxy server)...

[SimonBrandner]

I guess element could do that too (letting the user set the proxy server)...

I guess it could but feels like a bit of an overkill for GIFs...

[ShadowJonathan]

Maybe allow a direct call to giphy servers for now, but work towards a "define your own proxy" setting, and maybe go towards a (reduced) set of domains you can proxy HTTP requests with from the synapse server.

Though that latter action is maybe a bit risky, seeing that it effectively can transform a matrix server into a HTTP proxy, which isn't the goal and can be a security vulnerability.

[seanfarley]

Also, if I'm not mistaken, gifs that I recieve via a bridge (e.g. from signal, discord, etc) are rendered as a link instead of playing in-place.

[yajo]

Can't I give more than 1 thumb up to this issue? If I create a github pseudo-users botnet and fill it up of thumbs up would this get prioritized?

I feel like I'm dumb without gifs...

[J-tt]

@Yajo I added to the bounty to try and get someone to work on it, but seems like it's been stalled out pending review by @matrix-org/design and @matrix-org/product (see: https://github.com/matrix-org/matrix-react-sdk/pull/5814)

[ShadowJonathan]

I just want to give an extra data-point to this; Meta (Facebook) was ready to acquire Giphy, but was recently blocked by antitrust regulators, so I don't think relying on giphy at all for GIFs is a good idea, I recommend looking into either other open-source GIF hosting and indexing websites, or to take a similar approach to this as with stickers and emojis; put them in rooms.

[s0p4LiN]

Would not be better to use Tenor GIF instead of Giphy ? In my duplicate post, i said Google Chat is using GIPHY but this is wrong, it use Tenor GIF as keyboard gif.

[ShadowJonathan]

@n4rkip0d im not sure swapping Facebook for Google in this situation will make much of a difference.

That said, I think that, with this other site, a degree of choice could be made on the user side, in which the user can explicitly opt-in to these websites for the GIF selection UX.

However, I'd also like to see element provide a native solution to this, where users can collect or share their GIFs amongst eachother, adding tags or categories.

[torinreilly]

I personally do not think there should be a default gif search included in Matrix (or Element for that matter) that ties you to a specific GIF API or service. Instead, I think this is a good candidate for a widget (or widgets) so that each user can decide whether they want to opt in.

I have been recently migrating myself and my friends over to Matrix as the new spot for our group chats. While not all of them are technical, most can appreciate the principle of self-hosting an e2e chat server and having more ownership of their data. What they DID NOT appreciate is the lack of a proper GIF keyboard.

As a result, I have been working on the following gif keyboard for Matrix and testing it amongst my friends: torinreilly/matrix-gif-keyboard.

It works by overriding the sticker-picker widget and sending the animated gifs as stickers. You can host the server yourself and it can pull gifs from Tenor, GIPHY, or from a custom list of sites using DuckDuckGo (not sure if this is worth it as discussed here).

Here's the catch. The way stickers are implemented in the Element clients means the sticker URL has to be an MXC URI. In other Matrix clients like FluffyChat, stickers can have remote URLs (unfortunately FluffyChat does not support widgets though :/). So prior to sending the GIF it must be uploaded to a Matrix server. I am accomplishing by providing the GIF keyboard app server a Matrix access token to perform the uploads. This is fast enough to be fairly imperceptible by the end user, but it also exposes an endpoint that can be used to upload just about any image to your Matrix content repo! Not good! The only saving grace is the rate limiting that is supported by the content repository /upload endpoint.

Outside of requiring a login to the widget, I can't think of a good way to ensure this endpoint isn't abused. Currently, there only a few values that can be passed to the widget including username and the matrix server url, but nothing that can be used to ensure the user is authenticated. Passing things like the username and matrix server url could be a bit of security through obscurity, but it is easily faked so not worth implementing IMO. Looking for suggestions here.

If we could get the Element clients to support stickers with remote URLs it would remove the need for and upload in the first place, while saving us disk space and removing the auth headaches. It is also possible that we could get this to work using other room events, but I ran into a lot of trouble trying to send most type of events from a widget (m.room.message including images, etc). If anyone has experience/advice along those lines I would love to pick your brain.

In the meantime, please check it out if you are interested! I personally would not implement this in any kind of production for the time being. If you would like to try it out and don't want to host it yourself, I could let you use one of my hosted GIF keyboard widgets but only if you promise to not abuse my server too much!

Here are the key issues I'd love some feedback on: A better way to authenticate uploads to the content repository Does not work as a custom widget/integration Investigate if maintaining the DuckDuckGo API backend is worth it

Here is an example of how it looks on Element on iOS:

[SethFalco]

(Note, I'm not a maintainer, or even contributor for that matter.)

@torinreilly While I can appreciate your comment and approach to this, I think it's disregarding user-experience, no?

There are users who don't even understand what a file system is, or think their documents were deleted because it's no longer in their "Recent" in Microsoft Office anymore.

A lot of my friends are non-technical, they don't want to tinker with things or "learn the software". If it's not immediately obvious, or not intuitive, then it might as well not be a feature at all to them. With GIF keyboards kind of becoming a "standard" feature, I don't think it's ideal to make extra steps to access them.

I think it'd be reasonable to implement a GIF server natively, it's up to the user if they choose to use it or not. An option to turn off the UI for it would be great, or even better, a way to provide custom sources. I don't think the default experience should be through a widget though, which most people new to Element won't even understand immediately.

That being said, your project and widget look awesome. I especially love that you've considered solutions beyond Giphy and Tenor. A nice source to also consider could be Creative Commons Search?

[torinreilly]

@SethFalco Hi, Seth. Thanks for the thoughtful response. I think you are right on a lot of points here and a flexible native solution COULD be ideal provided it allowed enough choice to the end user. The thing I find nice about widgets is you are not beholden to the design decisions of anyone! If someone doesn't like how my widget is implemented or which services it targets they could use a different one. Or even fork it themselves and make it available to whoever they like.

Just to clarify, my intent is not for everyone to set up their own GIF keyboard server. In my case, I have a single app server that uses a bot account's token to perform the uploads to the content repository, but the message sent is actually on the requesting users behalf. Provided someone has setup a server, each user only needs to complete the one time setup of adding the widget in place of the sticker-picker described here: https://github.com/torinreilly/matrix-gif-keyboard#3-enable-widget.There is also the ability to add it through the UI as a "Custom Integration", which only requires the URL the app is hosted at (still working out some issues with this route for my app, it displays fine but messages are not sent).

I agree that extra step is inconvenient and might be confusing to some users, but I think most would not balk at it provided there was a sufficient guide.

I will definitely look at adding more search options. The DuckDuckGo API does allow for filtering results by license, so I was planning to implement that BUT I think Creative Commons Search would probably provide better results in the case of more open licenses! Thank you!

[captainperth]

I still can't believe we don't have a solution for this issue. Must I continue to manually download and upload gif files to truly express myself in element? Will someone please think of the memes!

[arkitoure]

I've just made the move from Discord/Telegram for our org over to Matrix/Element and while doing customisations we just realised using gifs 'natively' is not available.

This has created a lot of frustration for our users as one would expect this day and age and are looking for a solution.

The Giphy bot solution is unintuitive as users simply don't know what to search for to get an exact match of what they wish to express. Moreover, they often don't remember the command to trigger the search.

Naturally, we would want to see a gif icon next to the emoji and attachment icons in the UI which triggers a selector and search field.

I like the @torinreilly solution as it deals with the encrypted room issue that the giphy bot does not. So just on first glance I would be inclined to have this baked into the UX as mentioned.

This is a critical feature that cant be missing in todays world of chat/messaging apps. I will be looking at the code to see how I can contribute.

@torinreilly were there any security issues or like thoughts you had in implementing this solution?

[torinreilly]

@arkitoure There are definitely some security issues in my current solution. The way widgets are currently handled, there is no way to verify that the user of a widget is logged in to a specific Matrix server. In reality, there are only a handful of properties that are accessible to widgets (room name, username, etc) but any one of those could easily be faked in an http request so they really only provide security through obscurity. Any additional params you wish to include must be passed as part of the URL used when setting up the widget.

In my current implementation, the widget app server has the ability to upload files directly to the Matrix server for which it is configured. This essentially gives anyone with access to the widget server URL cart-blanche to upload files to the Matrix server (not good!).

This may be solved in a couple of ways:

First, I am hoping that the often discussed support for using a Matrix server as an OpenID source comes to fruition. If this happens we could easily verify the user via OpenID.

The second idea I had was to create a token or tokens that could be supplied as part of the widget URL during setup (ie https://widgeturl.com/?token=hdwry823hwekhr). This token could then be checked against a list of valid tokens by widget app server. This is better, but still relies on trusting users to not widely distribute tokens (might work OK for small communities or groups of friends) but for any large scale implementation the overhead of monitoring and revoking tokens used for abuse might not be worth it. (Additionally, I do not love the idea of passing the token as a URL param, but this is currently the only viable way to pass data to widgets as far as I know.)

The third option is to remove automatic upload to the server and require the user to copy and paste the GIF from the keyboard. I find this option to be less than intuitive and does not always work as expected, especially on mobile. It does however remove almost all of the risk of abuse by third parties.

The latter two solutions are less than ideal, but I could certainly add support for both of these with relative ease if there was demand.

[turt2live]

There are early designs for this up at https://www.figma.com/file/ifIvZA4XIP1JTyndHFr5Ub/Community-PRs?node-id=21%3A15691 however the technical undertaking of the proposed changes is a bit on the difficult side.

Specifically, it relies on stickers being an API surface we can call due to them being a widget - this isn't impossible, and is in fact planned under my widgets work umbrella, but not something we have today. The designs also appear to rely on power levels for this stuff, which is a bit difficult to enforce at the moment, but not impossible with the current structure and future capabilities like extensible events.

For a code review perspective: I don't think we can reasonably accept PRs on this until product and design have taken a much closer look, possibly with an angle of custom stickers/emoji too, as otherwise the code is likely to bitrot or change dramatically.

[seanfarley]

Why is this a server api at all? Just make a client-only keyboard that pastes a gif into the chat. I want to like matrix but y'all have a damn-near infinite feature matrix that makes progress nigh impossible.

[turt2live]

For simple support, sure, we could do it entirely client side. For future options of potentially paid sticker packs and similar though, there needs to be a server :)

[seanfarley]

For simple support, sure, we could do it entirely client side.

Then do that? Don't let perfect be the enemy of good.

For future options of potentially paid sticker packs and similar though, there needs to be a server :)

Dear god, stop.

[Jayddo]

I feel like there should be a workable solution for the very short term, as the simple gif search and paste has now become such a widespread colloquialism that its absence prevents wider adoption of the whole matrix platform.

(it might seem like an exaggeration, but it's not, and it's also not the focus point, the focus here is to be able to search & paste gifs with the same ease as other messaging platforms).

A better structure with a server for the long term is perfectly fine, but it's not needed now.

[turt2live]

this is why I've kept it in the design team's queue and not assumed their previous designs would work: we might not have the engineering bandwidth to fix stickers, but we may very well have the bandwidth to add gif searching.

For clarity: I was pointing out problems with the current designs, not proposing that the design team double down on the thing they've produced.

[ShadowJonathan]

I'm not sure why stickers are relevant in this issue? Imo, this issue explicitly (and only) calls for something akin to the /giphy command on discord, and the @gif bot on telegram, to have a simple lookup-and-paste for GIFs.

This can be achieved by client-only changes, while I have opinions on making stickers paid, I don't see how stickers are relevant at all to this issue.

[turt2live]

stickers and gifs are effectively the same thing at a technical level, regardless of presentation. As with most things Matrix, we don't generally want to tie ourselves to a specific gif provider so we'd probably lean on the sticker infrastructure a bit to anonymize access and generify the gif suppliers.

it's not a hard requirement.

[ShadowJonathan]

Alright, but then I just want to note that mentioning paid stickers brings in a very bitter point in the conversation that can further pollute and confuse it.

It was also framed in a way that stickers has to be a client-server api because it should become a paid offering, which strikes the wrong tone.

[turt2live]

it's no secret that stickers were always intended to grow a payments side to them - it would be entirely optional though. The challenge isn't the payment side though, it's that the current design mockups require gifs and stickers to be side-by-side with native UI: we'd need an API to be able to support this, which we don't have. Other options need to be explored, hence the call for re-design and product to be involved :/

[iceHtwoO]

How about the feature where a user can send the URL of a gif and the clients render it? This could be a compromise to make it a bit easier for the user and your not going to tie yourselfs to a particular provider.

[turt2live]

I think we generally want that for all images, yea. I can't find the issue easily though :(

it's a similar idea to inline widgets where things like youtube URLs magically become embeds (optionally).

[moodler]

Travis, like many others, I believe this issue to be very important for widespread adoption of Matrix.

I've had a conversation with Mathew Hodgson about it where he said the core team is busy and that "we welcome community contributions". Which is totally fair enough.

However, to now have issues like this effectively shut down with an over-complication and "the core team has to do it" at some possible unspecified time in the future is not great and counter to an open source ethos.

How can we work as a community here to get a gif picker done for all users? (You can always combine it with the sticker picker later)

IMO the gif picker is simply client-side, and the gif file should just be retrieved by the sender's client from the search service and sent as a file. This will work on all clients, and avoids any worries about tracking bugs etc.

[beposec]

I second that. Just a client side gif picker would help a lot for now. Currently the iOS App is not even not able to play GIFs. They are converted to png... That's not what a messenger 2022 should do.

For later it's sure good to implement a server side solution.

[ShadowJonathan]

The "GIFs are shown as PNGs" is an synapse as well as specification issue, see https://github.com/matrix-org/synapse/issues/1278 and https://github.com/matrix-org/matrix-spec/issues/453

[finncelta]

I 100% agree that this goes completely against what open source software should be. We have seen multiple PR about this exact issue and a lot of feedback on every request of GIF implementation that shows that users really want this feature. This issue has also been 7 years in the making and all we have to show for it are designs "we don't want to double down on", a temporary client side solution like @iceHtwoO has built seems totally adequate for the moment and would make many people in the community happy and might even help garnering new users.

[J-tt]

As someone who has a bounty on getting this issue resolved, seeing the core team drop its priority makes me genuinely sad, especially when others have clearly put effort towards claiming the bounty.

Is there any way to put money towards an issue to get it actually prioritised?

[finncelta]

I've been talking with @iceHtwoO during the creation of his fork and to now see it just get dropped is kind of sad, we've been talking about maintaining a working fork that has the GIF integration, kind of like the SchildiChat guys do it. Other than that I don't see a way to get this into element without the core team implementing it themselves or becoming more open to the idea of client side GIFs.

[turt2live]

Hang on, there is definitely room for the community to contribute this. The current designs are far too complicated for someone to contribute though, which is why I'm sending it back to their queue.

[turt2live]

In an effort to add clarity: all the discussion about stickers and mixing the infrastructure is to convey that the current designs are not feasible. We may very well explore it in the future, but for right now it's obvious we need something easier to implement and support.

With PRs stacking up against the feature, we can't reasonably do code review though, so have asked design to take a look at the general problem rather than any individual PR. Once that's done, anyone is welcome to contribute it - the core team might look at it from a higher, more generic, point of view, however.

[moodler]

Is there anywhere we can see progress or timelines from the core team on this and perhaps contribute to UX testing?

I could send thousands of users your way if needed.

That said, I don't think it's a complex UI at all. There are great examples to borrow from in WhatsApp, Twitter and Telegram.

[captainperth]

Is there anywhere we can see progress or timelines from the core team on this and perhaps contribute to UX testing?

I could send thousands of users your way if needed.

That said, I don't think it's a complex UI at all. There are great examples to borrow from in WhatsApp, Twitter and Telegram.

Seconded, I work in a small team that uses element for all internal messaging. We'd love to contribute if it means helping getting this feature off the ground.

[moodler]

So, current status is, as I understand it, that the product team will be working on a recommended UI spec for this seven-year-old popular issue and publishing it here.

At that point, any one of us in the community who wants this done can implement it, and assuming the code is good, it will be integrated quickly. OK, great.

Since this plan involves us (community) doing the bulk of the work and contributing resources, the respectful thing to do would be for someone from the core product team to publish some info on expected timelines. We have zero visibility into their queue and ideas. Is it a week? A month? A year? Another 7 years? ;-) Just let us know please, thanks.

Good communication makes good open source projects.

[beposec]

Bump! Any Update? @moodler asked important questions 😊