Due to upcoming changes in Emacs 29.1 utf-8-auto now produces a byte
order mark at the start of strings and no longer matches utf-8-unix.
Emacs is also not equipped to deal with byte order marks appearing on
their own, and will parse them as a symbol with the name \uFEFF (eek).
The end result is that trying to decode utf-8-auto with utf-8-unix in
Emacs 29 will result in an insanity inducing bug where calling read on
a string with a byte order mark (that doesn't render!!!) appears to
produce ... absolutely nothing, which turns out to be the symbol
mentioned above.
Given the absolute nightmare that byte order marks cause in Emacs, I strongly suggest switching everything over to utf-8-unix to avoid nasty cases like this in the future, and to save 3 bytes on every message!
nnicandro
commented
1 year ago
Due to upcoming changes in Emacs 29.1 utf-8-auto now produces a byte order mark at the start of strings and no longer matches utf-8-unix. Emacs is also not equipped to deal with byte order marks appearing on their own, and will parse them as a symbol with the name \uFEFF (eek).
The end result is that trying to decode utf-8-auto with utf-8-unix in Emacs 29 will result in an insanity inducing bug where calling read on a string with a byte order mark (that doesn't render!!!) appears to produce ... absolutely nothing, which turns out to be the symbol mentioned above.
https://github.com/emacs-mirror/emacs/commit/cfd2b3504ab5de6eb5f3c7a0784cb447883e1326
Given the absolute nightmare that byte order marks cause in Emacs, I strongly suggest switching everything over to
utf-8-unixto avoid nasty cases like this in the future, and to save 3 bytes on every message!