Emoji in Anchor editor deletes everything after and including Emoji

[adamgreenough]

Summary

When inputting an emoji in to the page or post editor, all the content including the emoji is deleted upon pressing save.

Expected Behaviour

The emoji will save in to the editor.

Actual Behaviour

Edited post, input Emoji, saved, everything after and including Emoji was deleted.

Context details (if applicable)

• Anchor version: 0.12.1
• Server setup: MediaTemple Grid
• URL www.adamgreenough.com

[adamgreenough]

Ok, I have managed to stop it deleting everything after an emoji. I assume this would be an issue for comments too, by altering the database encoding I have got it to stop deleting everything after an Emoji. Now it just seems to convert them to '????'. Still frustrating. :(

ALTER TABLEanchor_pagesDEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ALTER TABLEanchor_pagesCHANGEmarkdown`markdown TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL ALTER TABLE anchor_pages CHANGE html html TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL`

[daviddarnes]

While emojis are the hip new thing, this is quite the edge case. We need to probably support emojis in the GitHub markdown method, eg :poop: = :poop:

[TheBrenny]

Why don't we just convert it to a \uXXXX string? Shouldn't break the database and is solely dependent on the users character set.

[daviddarnes]

@TheBrenny it shouldn't, but which is an easier string to write? \xF0\x9F\x98\x81 or :grin:? 😉

[adamgreenough]

OSX, Windows 10 and mobiles all have emoji pickers now, I would opt for converting the unicode characters, particularly in comments where a visitor might naturally use an emoji.

[daviddarnes]

@xadz that's quite a limited amount of devices, we can't just make it work for the latest OSs only. I understand that we want to allow unicode, but we can't depend on that only. Same way that GitHub, WordPress and other platforms don't depend on them

[Radiergummi]

@daviddarnes considering you'd have to write a JS replacement script to insert emojis instead of the shortcodes anyhow, replacing the unicode strings with the icons only on older devices that do not support emojis natively would be the cleaner solution.

[daviddarnes]

@Radiergummi wouldn't inserting emoji <img> elements for all devices be a cleaner solution? This is how it's done for Jekyll and alike. It guarantees the emoji will be shown no matter what device it's on, without the need for JS anyway. The user could apply them with either the native keyboard or the shortcode, we would just need to make sure both string types are caught

[Radiergummi]

@daviddarnes Yup, you're right. I just checked how Facebook does it - they use <img> tags with the unicode representation in the alt attribute, which sounds like a sane solution to me...

[daviddarnes]

@Radiergummi great, thank you for checking other sources and contributing your point of view 😃

[TheBrenny]

@Radiergummi, but which source of emoji icons would we use? I guess we could ship our own emoji image set, but that would make anchor a real anchor in size? Is there a CDN for emojis as SVGs - maybe Emoji One? (Any other format would cause blurring)

[daviddarnes]

@TheBrenny we could use emoji one CDN, that's a good choice. We could use the default iOS ones too as I believe GitHub has a CDN for those too, they are pngs but they're massive so they'll only blur if you put them in a large heading or something like that.

[Radiergummi]

I'd say go for emoji one, as they're open source!

[daviddarnes]

@Radiergummi good point, Emoji One it is!

[adamgreenough]

Emoji One +1. This would be a great use case for plugins, could be something that is an option now though. Would be keen to see this implemented ASAP though, been experimenting but can't figure it out yet 🙈.

[daviddarnes]

You're right @xadz, this would be ideal as a plugin as to keep Anchor's footprint small

[TheBrenny]

Just thought of inserting another $2 coin here... Anchor claims to be ~250KB. The PNG of the EmojiOne Spritesheet (below) is 3.8MB. Simple math makes this PNG file (3800/250=15.2) roughly 15 installations of Anchor. What a :lollipop:!

[adamgreenough]

How far are we from plugins?

[daviddarnes]

@TheBrenny I think that's why you mentioned a CDN for it?

[daviddarnes]

@xadz not sure I'm afraid

[adamgreenough]

Also @TheBrenny while the emoji sprites do add a bit of overhead this wouldn't actually be in the anchor installation of course. I think while the size of anchor is great the power for me is in the flexibility. I've managed to tack on a lot myself but as part of Unicode and definitely here to stay I think supporting it at an editor level at least as an option would be great. :-)

[TheBrenny]

@xadz, like @daviddarnes said, if it get's implemented as a feature rather than a plugin, it'll be taken/borrowed/used/[insert word here] from a CDN network. Luckily there are a few CDN networks such as Rawgit that grab from GitHub and pass it on with production or development speed using MaxCDN's network. This would probably be the best way to implement, but again, that's assuming that this becomes a built in feature. Maybe when 1.0 drops, this will be a default plugin? Might be worth starting this on feathub to see how popular this request might become...

[adamgreenough]

Would it be possible to run this on the editor? Looks like the only other thing that would be needed would be the CDN which could be conditionally loaded

http://git.emojione.com/demos/latest/phptoimage.php

[ironpike]

If you decide to use CDN, please at lease make it configurable. Including GFWed services such as Google(JQuery) or CloudFlare will result in inaccessibility or disruption of the whole website for Chinese users.

[Radiergummi]

Concerning plugins... Once the infrastructure is up, I'd strongly favor a set of core plugins Anchor is shipped with. Emojis would be a great use case for this. As these plugins would be required to run Anchor, they can be depended upon without actually extending the base source, so they make no additional debugging noise. That is my opinion though, and maybe way out of scope of this issue ;)

[daviddarnes]

@ironpike if we do use a CDN, we won't make the Emoji assets render blocking, so don't worry.

@Radiergummi good point, this seems like a decent compromise. Shipping a "regular" package of Anchor with the option to remove such plugins to maintain the load speed and low footprint. Could even supply Anchor "vanilla" without any plugins.

P.S: This seems like an awful lot of discussion around emojis, sign of the times?

[TheBrenny]

I know I'm resurrecting a dormant issue, but I'm increasingly seeing the Emoji's worth. I'd love to use the same rules that GitHub uses when inserting emojis, as soon as you enter a : that is preceded by a /\s/ (RegEx), pop up a dialog that allows the user to enter the correct emoji. This could become very tedious very quickly, as we'd need to handle the images on the popup, and we can't really be sending POST requests for all the possible emojis. Perhaps we can ask the user for permission to grab all the emoji from... Wait, that's starting to sound like a plugin. :confused:

[daviddarnes]

I think this is ideal as a plugin, however we should provide some kind of fallback if a user places in an emoji from a keyboard shortcut. It's true that emojis are ubiquitous and becoming increasing useful, but I still think they are best bundled into a plugin

[TheBrenny]

The fallback could be when parsed from Markdown to HTML, we convert them to img tags with an emoji class? It'll then be up to the theme to make sure they're inline and at the correct sizes.

[daviddarnes]

Well I'd think the theme would let images inline to posts and general stay inline, the images would be the CDN right?

[TheBrenny]

Exactly. But I think for the most complete functionality (because this wouldn't work in the Admin area) it's best suited for a plugin. That way, this can get included into the admin area, without it being dependent on the currently selected theme.

[daviddarnes]

@TheBrenny good call

[frozenpandaman]

Just ran into this as well. Has anyone solved this?

For now I guess I can embed pictures from Emoji One.

[2xAA]

Emoji is part of Unicode though. Would it not be better in the long run to store the character values within the database, ensuring coverage of all characters entered 'ever'?

Unfortunately the fix for this is making sure you're using MySQL 5.5.3 and storing everything using utf8mb4, which could lead to some users breaking pretty hard.

Great article on supporting emoji within MySQL: https://mathiasbynens.be/notes/mysql-utf8mb4

[Radiergummi]

@2xAA considering it will take a few years until every visitor will have native emoji support, that can easily be changed with an update in the future, but currently, supporting legacy browsers/devices should have priority.
You're right about the database collation, though I suspect older MySQL versions won't be an issue once we have broad emoji support.

[2xAA]

Could we support legacy browsers but add in a MySQL version check to switch the charset if the version is > 5.5.2 since utf8mb4 is backwards compatible with utf8? Looks like an easy change in the install database files.

The legacy devices should be able to parse the emoji with a 3rd party library such as twemoji. Though I can't vouch for its compatibility on old browsers.

Best of both worlds!

As was said before, Emoji not being displayed is far more of a plugin issue I think.

Characters not being input into the database correctly though, that should be fixed as soon as possible.

We're dealing with two separate issues really.

[bmuessig]

I would suggest allowing the installation of the emoji pack on the own webserver itself (as an option). I don't like having to rely on an external service and my hosting package has tons of storage (20 GB) and unlimited bandwidth so I really don't care about < 5 MB of emoji which I can opt for caching anyways.

[daviddarnes]

I see your point @bmuessig, but not everyone has 20gb to play around with. Also, emojis do get updated quite often so if people self host them they're going to have to keep uploading packs to make sure it's up to date. It would be easier to keep the plugin up to date and have a single source for emojis

[bmuessig]

@daviddarnes I absolutely think there should be the option to outsource it, but I would love the option to host it on my own server. But with all this being kindly open source, i could just fork the plugin and add local emoji storage support :)

Edit: I am so sorry. Something with GitHub went wrong on my mobile and it posted the same comment multiple times.

[daviddarnes]

@bmuessig that seems like a fair conclusion 😄. We can add it as a future option to the plugin

[2xAA]

Should we start a new issue on encoding then? I feel we've diverged from the original issue which was to do with encoding.

Displaying emoji should not be a concern in this thread. That is a clientside issue, something for the host or client to figure out.

[daviddarnes]

@2xAA you're right, encoding and emoji support are two issues. Let's not go off topic!

[adamgreenough]

For anybody that stumbles across this thread looking to support emojis, it is quite straightforward to use GitHub style :shortcode: implementations with Emoji One with the following tools:

• https://github.com/Ranks/emojione
• http://git.emojione.com/demos/latest/class-convert.html

There are also classes to convert the Unicode chars to the Emoji images but that would first require resolution of the discussed encoding issue. Hopefully we can get that solved at least in the next release. :-)

[TheBrenny]

@daviddarnes @bmuessig, perhaps in the "Site Settings" of the Admin Interface, we have a textbox which states the location of the SVG/PNG of emojis to use? That way if it's local, you can throw a local URL in there, or the CDN? The only downside is that the server needs to collect the emoji image from the CDN rather than the client... 🤔

[bmuessig]

That sounds great!

----- Ursprüngliche Nachricht ----- Von: "Jarod Brennfleck" notifications@github.com Gesendet: ‎03.‎02.‎2017 17:53 An: "anchorcms/anchor-cms" anchor-cms@noreply.github.com Cc: "Benedikt Müssig" benedikt.muessig@gmail.com; "Mention" mention@noreply.github.com Betreff: Re: [anchorcms/anchor-cms] Emoji in Anchor editor deletes everythingafter and including Emoji (#1119)

@daviddarnes @bmuessig, perhaps in the "Site Settings" of the Admin Interface, we have a textbox which states the location of the SVG/PNG of emojis to use? That way if it's local, you can throw a local URL in there, or the CDN? The only downside is that the server needs to collect the emoji image from the CDN rather than the client... 🤔 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[daviddarnes]

@TheBrenny this could get complex quickly. The ideal would be a simple plugin that allows the use of shortcode style emojis in Anchor and use the emojione library for the images. Any further features, such as custom emojis and shortcodes, will just make the development process longer. Also the use of emojione means that as long as we attribute it correctly we can use without getting into legal bother