search

com-lihaoyi / upickle

uPickle: a simple, fast, dependency-free JSON & Binary (MessagePack) serialization library for Scala
https://com-lihaoyi.github.io/upickle
MIT License
707 stars 158 forks source link

Use UTF8 when reading json/msgpack bytes #464

Closed lihaoyi closed 1 year ago

lihaoyi commented 1 year ago

Fixes https://github.com/com-lihaoyi/upickle/issues/413

We already hard-code support for UTF-8 when rendering JSON and MessagePack https://github.com/com-lihaoyi/upickle/blob/6e2d5713bf745094207ad6ab6ac2b824a28a229c/core/src/upickle/core/RenderUtils.scala#L28 https://github.com/com-lihaoyi/upickle/blob/6e2d5713bf745094207ad6ab6ac2b824a28a229c/upack/src/upack/MsgPackWriter.scala#L166, so the only thing missing is to ensure we use UTF-8 when reading

Added some simple tests under upickle.jvm[_].testNonUtf8 running with -Dfile.encoding=US-ASCII exercise these code paths