nlohmann / json

JSON for Modern C++
https://json.nlohmann.me
MIT License
41.46k stars 6.59k forks source link

dump(): Non-conforming with JSON-spec escape of strings? #4088

Closed theIDinside closed 11 months ago

theIDinside commented 11 months ago

Description

This library seems to not be in compliance/non-conformant with the JSON spec, though I'm not 100% on that. None the less, it shows serious differences from almost any other library or "first-class citizen" of languages with built-in support of JSON (as is the case in Javascript, or Python for instance).

Reproduction steps

json obj;
obj["foo"] = 10;
obj["str"] = "\nhello world!\n";
obj["baz"] = 30;

const auto dump = obj.dump();

In javascript, if we create the following object and serialize it using JSON.stringify(obj) we get the following:

let obj = { a: 10, foo: 20, str: `
hello world!
`, baz: 30};

JSON.stringify(obj);

// we can also do 

let obj = { a: 10, foo: 20, str: "\nhello world!\n", baz: 30};

Python example:

obj = {}
obj["a"] = 10
obj["str"] = "\nhello world!\n"

import json
json.dumps(obj)

Expected vs. actual results

The outputs from the library is

// output
// "{ "foo": 10, "str": "\nhello world!\n", "baz": 30 }"

Where as Javascript produces

// output
// '{"a":10,"foo":20,"str":"\\nhello world!\\n","baz":30}'

and Python produces

'{"a": 10, "str": "\\nhello world!\\n"}'

Notice the correct escaping of the string. This library seems to only actually do the actual escape and not the "doubly" escape, which as far as I am aware, is not JSON-compliant?

I came across this particular bug (if it is a bug - it sort of feels like a bug at least?), because my application communicates with javascript over the "wire" (whether that is a socket, pipe, stdio, doesn't matter) - and fails, because nlohmann doesn't produce proper JSON string output that Javascript's JSON-functionality can parse.

I've used many languages before, using JSON as a communication protocol, and they all exhibit the same behavior, so I was under the assumption that this library would also be able to function in that manner. Perhaps there is some other feature in this library that produces the behavior seen above? (Python's json.dumps() for instance ,works in this way).

JSON spec says something like:

A string is a sequence of Unicode code points wrapped with quotation marks (U+0022). All characters may be placed within the quotation marks except for the characters that must be escaped: quotation mark (U+0022), reverse solidus (U+005C), and the control characters U+0000 to U+001F. There are two-character escape sequence representations of some characters.

Here's a stack overflow post discussing this issue, as well:

How do I handle newlines in JSON?

Minimal code example

No response

Error messages

No response

Compiler and operating system

clang++-15

Library version

v3.11.2

Validation

nlohmann commented 11 months ago

I am confident the escaping of this library is correct. There are thousands of tests and compliance checks, and also many projects where the library is used together with other libraries. Please check again if there is some error in your setup.

theIDinside commented 11 months ago

I'll check again. Could you please inform me where the tests are (if there are any) where .dump is verified to produce escaped escape-sequences (as per JSON spec and as Javascript & Python examples above), perhaps that will guide me in what I'm doing wrong.

Edit: It could potentially be that another library is simply eating the \\ when doing it's formatting (thus producing the incorrect \ instead. So if that test exists, it would be greatly helpful as this would conclude this issue entirely).

theIDinside commented 11 months ago

Yeah, I'm closing this. Another library was indeed eating that extra escape character.