search

TiddlySpace / tiddlyspace

A discoursive social model for tiddlers
http://tiddlyspace.com
Other
106 stars 38 forks source link

Including emoji results in error #1033

Closed bengillies closed 11 years ago

bengillies commented 11 years ago

To reproduce:

Try and save a tiddler with an emoji character (from the osx character viewer)

Expected:

It can save

Actual:

The server responds with an error

cdent commented 11 years ago

The version of mysql currently being run doesn't support 4bytes-per-character. Emoji is in that set. mysql 5.5 and beyond has a fix for it. There are plans, but not yet schedule, for upgrading to modern mysql (and python and debian). That same article mentions some workarounds (using varbinary) but I reckon the best bet is to put some schedule on the plan. However, it would be good to put a failing test in place now so things are worked on and to trap the error in a more friendly fashion. What error code did you get @bengillies? Was it a 500 or some 4xx?

One issue will be that column indexes (such as tiddler title) will need some adjustments because if you have 4byte wide characters the total index width of 767 means title lengths of 191 or using prefix indexes. (Mostly noting this for future reference).

bengillies commented 11 years ago

It's a 409 conflict error, which seems weird in itself.

cdent commented 11 years ago

409 is used when the representation of a tiddler PUT is invalid somehow. 412 when the headers are invalid. The title (which comes from the URI but is till part of the rep) is being considered no good.

So in this case the title you provided is (unfortunately) not valid, because it cannot be stored in the database: mysql doesn't think it is a valid unicode string. And until relatively recently that was true: The "planes" that emoji are in weren't used. Now they are.

cdent commented 11 years ago

see https://github.com/cdent/tiddlywebplugins.mysql/commit/384ef99622062b36ac92540294d036e0022ae486

cdent commented 11 years ago

This should be all fine now on tiddlyspace.com and in the latest versions of the relevant packages.