Closed mvayngrib closed 5 years ago
I recommend against other encodings.
@tbfleming thanks for the quick response! does that mean that there isn't actually a hard restriction and/or validation for utf8, and that other eos clients may use other encodings, which will break a utf-8 only TextDecoder? Do you know how the other implementations deal with this (eos-go, eos-...)?
the reason i ask is text-encoding
is a 500kb-heavy dep (!), a huge chunk of our React Native bundle. If all of those encodings aren't actually needed, we can inject a smaller Buffer.from/buf.toString()
based polyfill. However, I want to be 100% sure that those encodings aren't actually needed...
edit: Buffer.from/buf.toString()
based polyfill for utf-8-only support
There's a lot of utf-8 data on chain. e.g. transfer memo fields have content in a wide variety of languages
right, the question is, if there's one with non-utf-8 data (is this allowed?), will it break the client?
some refs from the cpp implementation (i'm not sure how to read these): https://github.com/EOSIO/eos/blob/master/libraries/chain/abi_serializer.cpp#L91 https://github.com/EOSIO/fc/blob/master/include/fc/io/raw.hpp
Nodeos, cleos, and most contracts are agnostic towards encoding. To them, a string is just a sequence of 8-bit bytes. Since both Linux and OSX use utf-8 as the default code page, cleos gets utf-8 support for free. Javascript has a more complicated story since its string type holds a variant of utf-16.
If you use a conversion which doesn't do utf-8, then non-ascii characters will get mangled, since others use utf-8 encoding.
@tbfleming thanks!
i see the defaults are initialized with utf-8 in mind, and the example in the docs does it similarly. Is support for non-utf-8 encodings needed? If this is specified in the docs, a link would be really appreciated. Thanks!
based on the git blame, maybe @tbfleming can help :)