Cannot deserialize the string end with '\0' whether is in lenient mode or not

[Warkeeper]

Issue Description

When deserialize a string which ends with '\0' ,it shows this error:

Caused by: com.google.gson.stream.MalformedJsonException: Use JsonReader.setLenient(true) to accept malformed JSON at line 1 column 30 path $
    at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1564)
    at com.google.gson.stream.JsonReader.checkLenient(JsonReader.java:1405)
    at com.google.gson.stream.JsonReader.doPeek(JsonReader.java:543)
    at com.google.gson.stream.JsonReader.peek(JsonReader.java:426)
    at com.google.gson.Gson.assertFullConsumption(Gson.java:904)
    ... 4 more

If set lenient to true, the error changes to:

Exception in thread "main" com.google.gson.JsonIOException: JSON document was not fully consumed.
    at com.google.gson.Gson.assertFullConsumption(Gson.java:905)
    at com.google.gson.Gson.fromJson(Gson.java:898)
    at com.google.gson.Gson.fromJson(Gson.java:846)
    at com.google.gson.Gson.fromJson(Gson.java:817)
    at com.unionpay.magpie.util.GsonTest.main(GsonTest.java:13)

How to reproduce it

Here's the code.

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class GsonTest {
    public static void main(String[] args) {
        String appleStr = "{\"color\":\"red\",\"weight\":200}\0";
        Gson gson = new GsonBuilder().setLenient().create();
        System.out.println(gson.fromJson(appleStr, Apple.class));
    }

    public static class Apple {
        String color = "red";
        int weight = 200;
        @Override
        public String toString() {
            return "Apple{" +
                    "color='" + color + '\'' +
                    ", weight=" + weight +
                    '}';
        }
    }
}

[Warkeeper]

I think it's pretty usual that a C client produces a json string which is ended with '\0', and the Java Client using gson to deserialize it. Is it possible to fix this issue?

[lyubomyr-shaydariv]

I think it's pretty usual that a C client produces a json string which is ended with '\0'....

I would say that it should not produce a \0-terminated payload, and the string you provided is actually a malformed JSON document, so Gson refuses to parse it.

Is it possible to fix this issue?

You can work around it by reading the JSON document via JsonReader, a reader that reads JSON tokens lazily:

final Reader reader = new StringReader("{\"color\":\"red\",\"weight\":200}\0");
final JsonReader jsonReader = new JsonReader(reader);
final Apple apple = gson.fromJson(jsonReader, Apple.class);
System.out.println(apple);

This does not fail because JsonReader reads the backing reader/input stream token by token, and once it consumes the final }, it suspends parsing not failing at \0. Once you ask it to parse/peek the next token with peek(), you'll get the syntax error exception:

...
jsonReader.peek();

A brief conclusion:

• It's most likely not a bug and Gson seems to be initially designed like that.
• It's recommended to use readers backed with input streams, not strings (deserializing the JSON document using StringReader is actually reading a string), for performance reasons.
• The JSON producer you're consuming produces invalid JSON documents and should be fixed if possible.
• You might also want to use FilterReader/PushbackReader to detect such malformed documents if you have to work with other libraries.

[Warkeeper]

Thanks for reply. @lyubomyr-shaydariv I can understand that Gson (with default configuration) refuses to deserialize a '\0'-ended string since it's not a valid JSON. However, when I set lenient to true, I may except that Gson will accept this kind of "JSON". Do I understand the lenient mode correctly?

[lyubomyr-shaydariv]

@Warkeeper To some extent. The lenient mode handles other special cases: https://github.com/google/gson/blob/ceae88bd6667f4263bbe02e6b3710b8a683906a2/gson/src/main/java/com/google/gson/stream/JsonReader.java#L295 (some of the items are weird IMHO). Your case is out of the list, but your issue is that Gson.deserialize (at least in the java.lang.String and java.io.Reader overloads) first deserializes the JSON document,and then checks whether the input document was fully consumed: the dangling \0 character is neither a valid JSON token (as jsonReader.peek() fails above right after the deserialization), nor the end of the JSON document.

[Warkeeper]

Got it.Thanks a lot. @lyubomyr-shaydariv For now I will work around it by using JsonReader , and maybe in the future I would make a pr to add this case for the lenient mode. This issue could be closed for now.