search

dmeranda / demjson

Python module for JSON data encoding, including jsonlint. See the project Wiki here on Github. Also read the README at the bottom of this page, or the project homepage at
http://deron.meranda.us/python/demjson/
Other
302 stars 76 forks source link

--allow=non-bmp #27

Open julienbaley opened 6 years ago

julienbaley commented 6 years ago

Hello!

Would it be possible to get a way to allow non-BMP characters? It is currently grouped together with non-portable issues, which is fair, but then I can't allow only non-BMP characters without allowing all the non-portable issues (like empty keys, for instance).

I'm happy to contribute it if you think there would be a chance it would be accepted.

dmeranda commented 6 years ago

I assume this is related to the portability warning produced by jsonlint when reading JSON that contains a raw (not escaped) astral plane character in a string value, e.g., the first warning below:

$ jsonlint myfile.json 
myfile.json:1:9: Warning: Strings containing non-BMP characters (U+101000) may not be portable
   |  At line 1, column 9, offset 9
myfile.json:2:12: Warning: Integers larger than 53-bits are not portable
   |  At line 2, column 12, offset 26
myfile.json: ok, with warnings

whereas using the --allow non-portable option suppresses the warning, but also suppresses other still-relevant warnings too,

$ jsonlint --allow non-portable myfile.json 
myfile.json: ok

First, for those that may read this issue... this should not be a problem if you are just reading JSON using the API, as when using the demjson module in your own Python application, nor when writing JSON. It is only a jsonlint issue where it tries to warn about valid, yet possibly non-portable data.

Yes, this is a fair issue to raise. Back when demjson was written, and this warning in particular, there were many JSON implementations which could not handle the full Unicode repertoire, especially without using the escaped surrogate pair syntax. Hence the non-portability warning. I believe that these faulty implementations may be much rarer today, so separating this particular warning out with its own option probably makes sense.

It probably warrants reviewing all the other non-portability warnings to see if any others are similar. I will look into this.

julienbaley commented 6 years ago

Thanks for the answer. I've submitted a PR for this particular case, but if you have a better approach, feel free to ignore. :)

julienbaley commented 4 years ago

Hi @dmeranda , I was wondering if I had any thoughts on this?

julienbaley commented 3 years ago

Hi @dmeranda ; it's now been a few years; any chance this could get resolved, or should I look elsewhere?