$ fluvio produce bin-test --raw -f test.bin
$ fluvio consume -B bin-test -O raw > test-output.bin
In the test-output.bin file, the bytes will be 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0xEF, 0xBF, 0xBD, 0x57, 0x6F, 0x72, 0x6C, 0x64, 0x0A. 0xF0, 0x90, 0x80 is replaced by 0xEF, 0xBF, 0xBD.
Steps to reproduce:
test.bin
with the following bytes0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0xF0, 0x90, 0x80, 0x57, 0x6F, 0x72, 0x6C, 0x64
. This is from https://doc.rust-lang.org/std/string/struct.String.html#method.from_utf8_lossy "Incorrect bytes" exampleIn the
test-output.bin
file, the bytes will be0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0xEF, 0xBF, 0xBD, 0x57, 0x6F, 0x72, 0x6C, 0x64, 0x0A
.0xF0, 0x90, 0x80
is replaced by0xEF, 0xBF, 0xBD
.I believe this happens to the inappropriate use of
from_utf8_lossy
in https://github.com/infinyon/fluvio/blob/29d1a11cdbb7976fc3e0c6934d2d981eb3af49ce/crates/fluvio-cli/src/client/consume/record_format.rs#L77This is binary data, it should not be parsed as utf8.