smheidrich
commented
1 year ago Thanks for the issue! Let me try to disentangle that a bit:
The byte string thing is a bit of a red herring, especially since you decode all the byte strings into Python strings anyway before passing them to json-stream. So a slightly simpler version to get the ValueError: No unicode character for code: 55356 at index 15 exception would be:
from io import StringIO
from json_stream import load
json_str = r'{"test": "\uD83C\uDFD4\uFE0F"}'
load(StringIO(json_str))["test"]AFAICT the actual bug is improper handling of chains of multiple \u... escape sequences in JSON. Single Unicode escape sequences, on the other hand, work fine:
json_str = r'{"test": "\u00C4"}'
print(load(StringIO(json_str))["test"]) # outputs 'Ä'Why is that? The underlying issue is that the JSON standard mandates that if one decides to escape Unicode code points beyond U+FFFF, it should be done using UTF-16 surrogate pair sequences, but json-stream's Rust tokenizer naively tries to parse each \u... escape into a Rust char separately, and Rust chars specifically can't represent Unicode surrogate code points. So that has to be fixed.
json-stream's pure-Python tokenizer appears to be able to parse them without raising an exception, but Python doesn't really seem to like the strings this produces:
from io import StringIO
from json_stream.tokenizer import load, tokenize
json_str = r'{"test": "\uD83C\uDFD4\uFE0F"}'
result = load(StringIO(json_str), tokenizer=tokenize)['test']
print(result)This will raise UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed in the print(result) line. I'm not sure whether these are technically correct or not, because the JSON standard does say
However, whether a processor of JSON texts interprets such a surrogate pair as a single code point or as an explicit surrogate pair is a semantic decision that is determined by the specific processor.
But I guess if Python strings, being pure UTF-8, can't contain these, then we should convert them to their UTF-8 equivalent anyway. So that should probably be fixed as well.
Finally, @goinnn, for your last point about quotation marks, that seems to be an issue with that wild chain of decoding and encoding you're doing rather than with json-stream:
>>> json_bytes = b'{"test": "Direkt auf dem Parkplatz der Gastwirtschaft \\"Neugebauer\\" zu finden, 24/7 \xc3\xb6ffentlich Zug\xc3\xa4nglich,\\r\\nLeckere Spei\xc3\x9fen und Getr\xc3\xa4nke von Donnersta bis S \\uD83D\\uDE99"}'
>>> json_bytes.decode('utf-8').encode('latin-1').decode('unicode-escape').encode('utf-16', 'surrogatepass').decode('utf-16'))
'{"test": "Direkt auf dem Parkplatz der Gastwirtschaft "Neugebauer" zu finden, 24/7 öffentlich Zugänglich,\r\nLeckere Speißen und Getränke von Donnersta bis S 🚙"}'The double quotes inside the JSON string lost the backslash that escaped them, so the JSON string does in fact end before Neugebauer then. Since json-stream yields that string before parsing further, it will only raise an error due to the now-malformed JSON if you request the whole dictionary, e.g. by doing list(load(...).items()) instead of load(...)["test"].
goinnn
daggaz
json-stream should accept byte string because we can not parse some JSON with json-stream but we can parse them with json library.
Using json library works:
Using json_stream library does not work:
We can do something like this:
But this does not work with another byte strings:
But this works perfectly with json library:
Also there is another relational problem. The original json_bytes variable contained quotation marks:
If I run the solution, json-stream cut the result: