Custom emoji support

[aumetra]

General design idea:

• New configuration section called instance
  • This section contains custom_emoji, description, name, etc.

The custom emoji type could look like this:

let CustomEmoji = { name: Text, path: Text }
let CustomEmojis = List CustomEmoji
in {
     CustomEmoji, 
     CustomEmojis
}

The custom emoji would then be mounted in some media sub-path, something like /media/custom-emoji/:emojiName, and would then, as shown in the path, indexed by their name
Not sure if this might be something that'd be better kept in the database, since this might need to synchronise between multiple nodes.

This is just a first design draft, completely open for discussion

[aumetra]

Note that this also needs database schema changes and new ActivityPub types

[aumetra]

In case anyone wants to take this issue, I'm willing to mentor through the process.
ActivityPub, especially these less documented parts of it, can be overwhelming at first.

[aumetra]

Support to load emoji used inside an object was added #110

[zeerooth]

Would remote emojis be something that we'd like to support? That's very much feasible to do and IMO it'd be quite a cool and useful feature, seeing how often emoji packs are duplicated between instances and how sometimes missing out on emojis not present on your current instance sucks.

The downside would be that potentially malicious actors could pull offensive emojis from some other server and disrupt the local instance, so it's necessary to add endpoints moderating the federated emojis + configuration variable for blocking that feature.

Mastodon issue reference: https://github.com/mastodon/mastodon/issues/8649

[aumetra]

Remote emoji would be interesting but it'd be a challenge of how to actually do the discovery of these emoji.

Do we store them when receiving them? Do we allow a mention syntax (ala :blob-dj@my-instance.example:)?

In case we use this mention syntax, how do we discover them? Do we just assume each remote server with custom emoji implements the Mastodon API? What about Misskey/IceShrimp?

[aumetra]

Another much more basic question, how do we configure these custom emoji? Configuration file? Datababse?

[zeerooth]

Another much more basic question, how do we configure these custom emoji? Configuration file? Datababse?

I was thinking of doing it through database. Have a new table for storing emojis that refers to media_attachments and a new service for adding, modifying and removing emojis. Eventually when we have a new kitsune-cli we can provide a way to add emojis through cli or maybe have a graphql endpoint and then hook it up to UI in the future.

I don't think using a configuration file for that is a good idea because we'd need to restart kitsune just to modify emojis which is really not needed.

Remote emoji would be interesting but it'd be a challenge of how to actually do the discovery of these emoji.

Do we store them when receiving them? Do we allow a mention syntax (ala :blob-dj@my-instance.example:)?

Yes, that's what I was thinking of, mention syntax would be great, but I still need to see if other fedi software works with that, especially mastodon of course. Otherwise there needs to be some translation layer. That is, the user picks emojis using :blob@example.com: but we use a normal :emoji: syntax in all the interactions with other AP servers and under tags we use emojis from different servers with original ids, like so:

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    {
      "Emoji": "http://joinmastodon.org/ns#Emoji",
    }
  ],
  "id": "https://example.com/@alice/hello-world",
  "type": "Note",
  "content": "Hello world :blobcat: :blahaj:",
  "tag": [
    {
      "id": "https://site11.example/emoji/123",
      "type": "Emoji",
      "name": ":blobcat:",
      "icon": {
        "type": "Image",
        "mediaType": "image/png",
        "url": "https://site11.example/files/blobcat.png"
      }
    },
    {
      "id": "https://site22.example/emoji/000",
      "type": "Emoji",
      "name": ":blahaj:",
      "icon": {
        "type": "Image",
        "mediaType": "image/png",
        "url": "https://site22.example/files/blahaj.png"
      }
    }
  ]
}

Shortcode collisions would have to be resolved by adding a number, prefix, domain or sth but it's definitely doable. Would that work? Again, I have to test.

In case we use this mention syntax, how do we discover them? Do we just assume each remote server with custom emoji implements the Mastodon API? What about Misskey/IceShrimp?

Yes, I actually started writing code assuming servers use mastodon API in which case I wanted to give users an option to choose a federated server and pull emoji packs from there. To reduce the possible spam of emojis (think of thousands of users pulling emojis from hundreds of instances) these emojis could be enabled, through preferences only to selected users (and any others that enable them through settings). Admins could also make emoji packs global for every user.

However, this indeed would only work for mastodon. I guess there are already so many mastodon servers full of unique emojis that it should satisfy users, but another approach would be to save every emoji that we pull through federating and fetching actors and notes and then allow users to choose these or for admins to enable some of them globally. The advantage of that approach is that it should work with any other servers that puts emojis as a tag type, so I think misskey does it too and a lot of other AP implementators, but the downside is that there's no way to pull any emojis that the server doesn't yet know about.

Which approach do you think would be better? If any?