Entry fails to save when embed field contains an instagram image with emojis in the caption

newism / craft3-fields

Address, telephone, email, embed and gender fields for CraftCMS 3.x

Other

54 stars 16 forks source link

Entry fails to save when embed field contains an instagram image with emojis in the caption #22

Closed dohayon closed 6 years ago

dohayon commented 6 years ago

leevigraham commented 6 years ago

Interesting… I'll check with @brandonkelly how craft handles emojis

brandonkelly commented 6 years ago

@leevigraham Craft emoji support has 3 facets:

craft\helpers\StringHelper::encodeMb4() will HTML-encode all 4(+)-byte characters. Only con is that they get converted to HTML entities, which may not be desired if the string isn’t HTML.
LitEmoji\LitEmoji::unicodeToShortcode() will convert most emoji to :shortcodes; and shortcodeToUnicode() will convert :shortcodes: back to the unicode emjoi chars. Only con is that any unsupported emoji characters will not be touched, which could still lead to errors.
PostgreSQL has no trouble with emoji, so if you have a site that needs good emoji support, you should use that if at all possible instead of MySQL.

leevigraham commented 6 years ago

Sounds like craft\helpers\StringHelper::encodeMb4() is the best option here given the output will always be html.

leevigraham commented 6 years ago

Example URL: https://www.instagram.com/p/BfYPK64FFe6/?taken-by=leevigraham

leevigraham commented 6 years ago

Looking at the craft code it looks like fields are using encodeMb4() in the fckeditor plugin

if (Craft::$app->getDb()->getIsMysql()) {
    // Encode any 4-byte UTF-8 characters.
    $value = StringHelper::encodeMb4($value);
}

The EmbedModel implements JsonSerializable which means it's skipped by Crafts serializeValue method…

Interestingly if you try to submit an emoji in a page title then the same issue occurs:

SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xF0\x9F\x98\x8Afu...' for column 'title' at row 1
The SQL being executed was: INSERT INTO `content` (`elementId`, `siteId`, `title`, `field_gender`, `field_address`, `field_email`, `field_embed`, `field_personName`, `field_telephone`, `field_telephone2`, `dateCreated`, `dateUpdated`, `uid`) VALUES (89, 1, ' 😊fun', NULL, NULL, NULL, NULL, NULL, NULL, NULL, '2018-03-01 10:27:50', '2018-03-01 10:27:50', '60cbea1b-8b6b-430f-869c-64cc95e8fd4f')

leevigraham commented 6 years ago

@brandonkelly what is Craft's position on emoji support? If support is only guaranteed on a DB level then I'm tempted to follow suit.

leevigraham commented 6 years ago

See:

leevigraham commented 6 years ago

@dohayon, I've implemented LitEmoji encoding / decoding. Let me know how the latest version goes

leevigraham commented 6 years ago