Implement sticker packs

rocurley commented 7 years ago

I'd like to add support for sticker packs to Riot. Since this would involve writing a lot of code, I wrote this little document up so we could discuss it before I've spent a long time on it. That said, while I'd love to have help, I'm happy to implement this myself.

Why sticker packs?

Most of my friends use Telegram as their chat client of choice. This isn't ideal for several reasons, and I'd like to get them to switch. It may seem silly, but I'm pretty sure the biggest lock-in factor there is stickers. Telegram's sticker system is really slick, and using anything without it or something similar feels like a real downgrade. Relatedly, I've framed most of this in comparison to Telegram's sticker system.

What makes sticker packs good?

I'd claim that the most important features are:

User creatable: This is nice on several levels. It saves work on the dev side, obviously. But more than that, this lets people make stickers for their specific interests.
Easily sendable: Stickers won't get used if it's not convenient to do so.
Easily copyable: Almost all the stickers I use, I have because I saw one I liked, and I added it to my collection. Telegram makes this process trivial.
Grouped: Part of the reason stickers spread so effectively is that if I see a sticker I like and add it to my collection, I get a bunch of other stickers along with it that I will likely enjoy.

The ideal outcome is an environment where stickers can be easily created, and easily spread between people when they chat with each other.

What are features we could do without?

I'm not super confident here, but:

Updating packs. When the creator of a sticker pack adds new stickers to it, that change could be propagated out to users who have the pack, or not. I'm inclined towards not supporting that, mostly on implementation difficulty grounds. One could imagine having sticker packs label themselves as replacements for other ones, and make it easy to replace the old one if the user adds the new sticker pack, but that probably doesn’t need to be in the initial version.

Implementation

I’m proposing a specific implementation here as a proof of concept, but I’d be happy to see it improved. Mostly I’d like to get some discussion about this so I don’t spend a long time implementing this and then have it rejected.

The design goals with the implementation were:

Work entirely within within matrix: no external servers.
Avoid changes to the matrix protocol.
Avoid requiring the homeserver to know about stickers.
Avoid nonstandard messages.

This design achieves the first three. To allow subscribing to sticker packs, the fourth is compromised by adding a custom field to the image message when stickers are sent.

A sticker pack is an archive containing one or more images (stickers), and a metadata file. The metadata would associate each sticker with zero or more emoji, and contain the name of the sticker pack. It would also contain whatever information I forgot was important. Since it's likely we won't know ahead of time what information is important, we can version the spec and save the version in the metadata file. It’s unclear what file formats should be supported: that probably depends on how inline images shakes out (although this can be done with the standard image sharing until then).

To publish a sticker pack, one would:

Create this archive (ideally assisted by the client, but it could start out as a manual process).
Upload the sticker pack to your homeserver. The resulting MCX URI enables other users to subscribe (a misleading term, since it's impossible to update a sticker pack) to the sticker pack.

To subscribe to a sticker pack, the client would:

Download the pack using the pack's URI.
Save the URI in the client config. This makes subscribing to a sticker pack a a per-user, not per-device affair.

When using the pack, the client would:

Check to make sure all the sticker packs in the client-config exist locally. This includes deleting sticker packs that have been unsubscribed from.
Allow the user to easily browse and send stickers in the sticker packs.
When sending a sticker, include in the message the sticker pack URI. This allows other users to preview and subscribe to other sticker packs.
(This can be added late) If the user starts a message with an emoji, show all stickers indexed by that emoji.

Stages

I think the natural way to break this down into sub-tasks, each of which can be implemented separately, is:

Implement sending (manually created) sticker packs from local storage. This is somewhat similar to Gregoor’s proposal for issue 2648. I’d be happy to coordinate with them on this.
(This doesn’t block anything else, it just has to come after 1) Implement a UI for creating sticker packs.
Synchronize a user’s sticker packs via the client-config.
When using a sticker from a pack, attach the link to the sticker pack.
Make a UI for subscribing to sticker packs you’ve seen.

tessgadwa commented 7 years ago

Let's do it.

See my comments at https://matrix.to/#/!blHeuKmYNNfIqDlWGO:matrix.org/$14844481771476529VOaQj:matrix.org

rocurley commented 7 years ago

Some details that tessgadwa and I hashed out over chat (it's all at the link above, so you can just read it there if you prefer). Stickers should be 512x(<=512) vice versa (one dimension 512, one <=512) pngs, or vector graphics (svgs?). Stickers can be associated with emoji. This implicitly requires the ability to recognize emoji. it seems simplest to piggyback off of the Emoji One conversion process (where if you get an emoji and nothing else in a message, it turns it into a bigger emoji). We also discussed making some number of default sticker packs, which tessgadwa is interested in doing (in addition to UI work for steps 2 and 5). Stickerpacks of about 12 stickers, ideally including ones that convey: happy, bored, sad, angry, curious, celebratory, congratulatory, amused. (This was pretty off the cuff though).

rocurley commented 7 years ago

Some more discussion here. We're pretty much set for stage 1 as described above.

It was suggested that we find an existing sticker pack format and just use it. The only such thing I could find was apple's here. I'm sort of disinclined from using it, because it supports a bunch of features we probably don't want (animated stickers, for example), as well as not supporting some features it should (emoji). I think a simple json format along the lines of:

{
    "name" : "Goats with sunglasses"
    "author" : "Nunya Business"
    "stickerpack spec version" : 1.0
    "stickers" [
        {
            "file" : "evilGoat.svg"
            "emoji" : "😈🔥"
        },
        {
            "file" : "similingGoat.png"
            "emoji" : "😊"
        },
        {
            "file" : "nondescriptGoat.png"
            "emoji" : ""
        }
    ]
}

would be a lot easier.

We discussed where to store packs, with the tentative plan to use mxc for now. Other options include IPFS, some website, or torrent hashes. We probably want to have an external website for searching for sticker packs as well at some point.

tessgadwa commented 7 years ago

This works for me.

ara4n commented 7 years ago

how are the emoji fields meant to be used here?

rocurley commented 7 years ago

If the user starts a message with an emoji, show all stickers indexed by that emoji. Since stickers are used more or less in the same context as emoji, showing a relevant sticker when you type an emoji can be pretty useful

tessgadwa commented 7 years ago

I was also noticing that when I used the built-on Riot emojii, I only get 8 emojii showing up. Is this standard?

tessgadwa commented 7 years ago

Potential artist to contact suggested: http://www.deviantart.com/art/Phil-The-Owl-629856759

tessgadwa commented 7 years ago

Good discussion about in Riot main room today about image uploading vs sticker packs.

Goal is to allow users to access a common set of sticker packs direct from the app without having to upload their own each time from the hard drive.

tessgadwa commented 7 years ago

Another big goal: Create a way for artists to upload/share stickers under a CC license for inclusion in Riot.

rocurley commented 7 years ago

Currently thinking we should just base64 encode the images and stick them in the json file directly. It's a bit inefficient (4/3), but that way we don't have to add a dependency on some tar library or store every sticker separately on the server.

ara4n commented 7 years ago

@rocurley the disadvantage of doing that is that you don't end up with a canonical URL for the individual images, which will encourage clients to go and take the base64 out of the JSON and keep uploading them in duplicate to their HSes. Ideally the HS should deduplicate the content, but currently synapse doesn't sadly. Whereas if you reference the actual bits of artwork as separate mxc:// URLs, they will automatically get deduplicated. And clients won't have to mess around re-uploading when they send 'em?

rocurley commented 7 years ago

@ara4n That's definitely a benefit of storing them separately. My main concern there is individual stickers getting deleted. If you just have one file, that's less likely to happen. I still don't have a great model for how likely files are to be deleted though.

rocurley commented 7 years ago

Also worth figuring out: How do we cache stickerpacks locally? I was planning on fetching them from the homeserver at startup every time for now, but I don't love that option. I looked into localstorage, but 5Mb isn't very much.

ara4n commented 7 years ago

I'd just cache stickers using normal HTTP caching in the client (at least for riot-web); it should work fine :)

In terms of content being deleted - there's always going to be a risk that the source content is deleted, whether it's a master copy or one of a million duplicates. For now I'd completely ignore it; practically speaking folks in Matrix hardly ever expire MXC content (and even if they do, it's only their remote caches), and in future switching to something like ipfs will make it go away anyway.

rocurley commented 7 years ago

Alright, sgtm.

tessgadwa commented 7 years ago

Aight.

On Fri, Jan 20, 2017 at 1:23 PM, Roger Curley notifications@github.com wrote:

Alright, sgtm.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vector-im/riot-web/issues/2950#issuecomment-274183468, or mute the thread https://github.com/notifications/unsubscribe-auth/AGLNG4ug22IJ86lrUNB3qND-p9ihCvA4ks5rUSXTgaJpZM4Ljr0n .

--

Tess Gadwa

Chief Executive Officer

Yes Exactly, Inc. | yesexactly.com | 413.325.8251

ara4n commented 7 years ago

one thought that could help here: if we switch to naming mxc urls based on content hashes then deduplication becomes trivial, as does fixing up and recognising content if the original content does get somehow deleted... see also https://github.com/matrix-org/matrix-doc/issues/446 and https://github.com/matrix-org/matrix-doc/issues/539

ara4n commented 7 years ago

@rocurley did this get anywhere?

rocurley commented 7 years ago

Sorry, haven't had much time on this for the past couple weeks. As it stands, you can send stickers if you've created and added them manually (with artisinal hand crafted http). Working on the UI for making sticker packs now.

tessgadwa commented 7 years ago

Cool. Let me know when the UI is ready.

rocurley commented 7 years ago

@ara4n so far we have code to send stickers from packs. The mechanisms for adding packs (making your own or adding packs you've seen) are still a work in progress. I'd like to get some code review before then, before we build too much on something that gets heavily modified. The issue is that it's not really mergeable at the moment: We'd be adding a sticker button that essentially nobody can use. It won't really be usable until we have the sticker pack creator working, or a set of sticker packs that are included by default (although that would be much easier to create if we had the sticker pack creator working). Idk what's the best solution here: I could make a mock PR?

tessgadwa commented 7 years ago

@rocurley where in the repo would one upload sample stickers to be used in testing?

rocurley commented 7 years ago

@tessgadwa Honnestly probably the easiest thing would be to send them over riot (to generate the mxc urls). That would make it pretty easy to put together a pack.

tessgadwa commented 7 years ago

Hey @rocurley I sent you some raw materials. Probably not what you're looking for but if you need specific image files there are plenty more under CC license (no attribution) on Google Images.

aviraldg commented 7 years ago

@tessgadwa Autocomplete is configured to only show the 8 most relevant ones, but there are more.

tessgadwa commented 7 years ago

@aviraldg how are we determining relevance?

aviraldg commented 7 years ago

@tessgadwa Fuzzy string matching, and now substring matching. I will be looking into adding an emoji picker later, which could probably handle this as well. In any case, I've been involved with rewriting the composer so I could probably help on the UI parts of this issue.

tessgadwa commented 7 years ago

@aviraldg that's fantastic. Does the code currently reside in the main Riot repo?

I also think it may be helpful to show stickers by Category. But really I think either or both could work.

lukebarnard1 commented 6 years ago

@rocurley, how close are we getting mxc URLs to stickers to Riot? The new message composer would need modifying such that users can search for stickers, but we should be landing some changes that allow for sending arbitrary <img src="mxc://..."/> in messages soonish.

lboklin commented 6 years ago

Any progress on this?

lukebarnard1 commented 6 years ago

@lboklin the Riot devs have been fairly busy with Communities over the last few weeks and so haven't had the bandwidth, but we'll be getting to it Soon.

rocurley commented 6 years ago

I kind of dropped this a while back, after I wasn't able to figure out how to get code review on what I had thus far (I don't want to throw stones here: I'm sure everyone's busy, and I'm not super familiar with the processes here). If we can get code review of the work so far, I'd like to keep working on this. I can't guarantee fast process (a lot of my time is spent job-searching right now), but I'm still interested in this feature.

ara4n commented 6 years ago

hm, i may be going insane, but I don't think i've ever seen any code for this? Is there a PR or a branch somewhere I should know about?

rocurley commented 6 years ago

https://github.com/rocurley/riot-web/tree/develop https://github.com/rocurley/riot-web/tree/develop

It's been a long time and TBH I have no idea how I tried to communicate these to you.

iav commented 6 years ago

@rocurley you have to press a button "new pull request" on top of main page of your fork on github.

arpsyapathy commented 6 years ago

Hi! Is there any test Riot version with the opportunity to exchange stickers? Or still not implemented?

lukebarnard1 commented 6 years ago

@psyapathy It hasn't been implemented yet, with the main devs working on stabilising widgets and implementing communities.

arpsyapathy commented 6 years ago

@lukebarnard1 Sad... And then there is the implementation of the Emoji picker on click? To click on the button and get a full list of Emoji?

lukebarnard1 commented 6 years ago

@psyapathy there's an issue for that somewhere. This isn't the place for discussion though, better off poking us in #riot

ara4n commented 6 years ago

So the original PR seems stuck, but we're hoping to get to this as part of Widgets in the relatively near future. Going to keep this bug open for historical value, and to act as a tracker for the upcoming stickerpack widget.

Qwertie- commented 6 years ago

I noticed there was some sticker picker integration in riot recently. Are there new updates to this issue?

lboklin commented 6 years ago

The integration seems to have been implemented in develop, but no actual sticker button appears when it's added.

t3chguy commented 6 years ago

Did you enable it in User Settings > Labs?

lukebarnard1 commented 6 years ago

Stickers have landed as a "Labs" feature on the develop branch.

@lboklin you're better off asking in #riot:matrix.org or opening a new issue. But yes, as @t3chguy says, make sure you've enabled the feature flag in user settings.

lboklin commented 6 years ago

I had not! It works after enabling it there.

lampholder commented 6 years ago

Okay, so we now have stickers in Riot/Matrix :)

What's the delta (from a user's perspective) between our impl and this original discussion?

User creatable: not trivially. Users can create stickers and send them to us to host, or host them on their own personal Dimension instance, but that's not something everybody is going to be capable of/interested in doing/willing to do
Easily sendable: :heavy_check_mark:
Easily copyable: not really. When you see a sticker somebody else has posted, there's not an easy link back to adding that stickerpack to your own collection
Grouped: :heavy_check_mark:
Updating packs: :heavy_check_mark: We get this 'for free' with the chosen impl (for adding new stickers, not modifying existing stickers in place)

Qwertie- commented 6 years ago

Is there any plan for user creatable packs? The way telegram does it is through a bot you message but I imagine creating a webUI or a custom page in riot for it might be easier to use.

turt2live commented 6 years ago

User-created (custom) sticker packs are in the proposal stage at the moment: https://github.com/matrix-org/matrix-doc/issues/1256

martindale commented 5 years ago

I don't think this should be included in Riot by default, but rather made available as a plugin.

element-hq / element-web