oxigraph / json-event-parser

JSON event parser is a simple streaming JSON parser and serializer implementation in Rust.
Apache License 2.0
9 stars 2 forks source link

Handle UTF16 lone surrogates #1

Closed bmeck closed 3 years ago

bmeck commented 3 years ago

JSON uses 2 byte hex encoding for unicode escapes. This can cause lone surrogate problems in UTF-8 encodings that need 4 bytes ( see https://unicode-example-characters.glitch.me/ ). For example:

πŸ”₯ can be encoded as "\ud83d\udd25". 🚸🚯 can be encoded as "\ud83d\udeb8\ud83d\udeaf".

Currently attempting to decode these results in errors:

    let json = b"\"\\ud83d\\udd25\"";
    let mut reader = JsonReader::from_reader(Cursor::new(json));

    let mut buffer = Vec::new();
    reader.read_event(&mut buffer)?;
panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidData, error: "Invalid encoded unicode code point" }', src/main.rs:86:15
Tpt commented 3 years ago

Thank you for this bug report! I should have indeed added support for surrogate pairs. I have just implemented them and released a v0.1.1 version. Your test now passes. Thank you again!

bmeck commented 3 years ago

Thank you so much!