Writing "raw" data using JsonWriter

[NicolaSpreafico]

Hi, I have a flow which read an image from a File, convert it to Base64, put the value inside a JSON and then write the data inside an OutputStream (which will send content using an HTTP Connection).

Due to image size, I can't read the whole image bytes in memory, then fully convert it to base64, then put it all the base64 value inside the json and create the connection.

So I'm using a JsonWriter in order to read a chunk of data, convert it to base64, write inside the outputstream and the repeat for the whole content of the image. Here is concept code:

@Override
public void writeTo(OutputStream out) throws IOException {
    try (JsonWriter jsonWriter = new JsonWriter(new OutputStreamWriter(out, "UTF-8"))) {
       jsonWriter.beginObject();
       jsonWriter.name("image_base64");

       // Read the image file with buffer
       Encoder encoder = Base64.getEncoder();
       ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
       ReadChannel reader = client.reader(... opening reader for file...);

           // TODO1
       jsonWriter.beginString();

       while (reader.read(bytes) > 0) {
           bytes.flip();
                 // TODO2
           jsonWriter.appendStringValue(encoder.encode(bytes.array()));
           bytes.clear();
       }
           // TODO3
       jsonWriter.endStringValue();

       jsonWriter.endObject();
    }
}

Because the JsonWriter only provides the method to write an entire String, for the moment the only way I found is reading the whole image, convert it to a full String and the call a single method. As mentioned in the opening, I need to do this processing with a buffer in order to not having alla the image bytes in memory. Please note the 3 TODO parts, which I'm not sure how to implement.

Is there a way to write "raw data" in the OutputStream which simply writes the characters I give to it instead of writing the exact String i give, sorrounded by "?

For the moment I totally ignored the JsonWriter implementation and I'm writing the whole content as raw data:

@Override
public void writeTo(OutputStream out) throws IOException {
    String init = "{\"image_bytes\":{\"";
    out.write(init.getBytes(Charset.forName("UTF-8")));

    Encoder encoder = Base64.getEncoder();
    try ( OutputStream outB64 = encoder.wrap(out)){
        ReadChannel reader = client.reader(... opening reader for file...);
        IOUtils.copy(Channels.newInputStream(reader), outB64,64 * 1024);
    }

    String end = "\"}";
    out.write(end.getBytes(Charset.forName("UTF-8")));
}

Please note that this is a simplified version, the real json also have additional values as metadata for the image and other stuffs

[NightlyNexus]

Is there a way to write "raw data" in the OutputStream which simply writes the characters I give to it instead of writing the exact String i give, sorrounded by "?

Are you looking for JsonWriter#jsonValue?

[NicolaSpreafico]

Yeah, I already tried that method but I got an error, I don't know if I'm using that wrong or it does not apply to my case. Here is an example:

The json I'm trying to get:

{
  "key": "foobarbaz"
}

And a simple main to test it:

public static void main(String[] args) throws IOException {
    String[] parts = {"foo", "bar", "baz"};

    try (JsonWriter writer = new JsonWriter(new PrintWriter(System.out))) {
        writer.beginObject();

        writer.name("key");

        writer.jsonValue("\"");

        for (String part : parts) {
            writer.jsonValue(part);
        }

        writer.jsonValue("\"");

        writer.endObject();
    }
}

Note that I created an array of "chunks" to simulate the string value.

Here is the error:

{"key":"Exception in thread "main" java.lang.IllegalStateException: Nesting problem.
    at com.google.gson.stream.JsonWriter.beforeValue(JsonWriter.java:641)
    at com.google.gson.stream.JsonWriter.jsonValue(JsonWriter.java:435)
    at TestApp.main(TestApp.java:21)
    Suppressed: java.io.IOException: Incomplete document
        at com.google.gson.stream.JsonWriter.close(JsonWriter.java:544)
        at TestApp.main(TestApp.java:27)

[NicolaSpreafico]

Here is a very ugly (but working...) implementation.

public static void main(String[] args) {
        try {
            String[] parts = {"foo", "bar", "baz" /* this array simulates all the chunks of data to write */ };

            try (MyJsonWriter writer = new MyJsonWriter(new PrintWriter(System.out))) {
                writer.beginObject();

                writer.name("key");

                // Fake value in order to make the json structure balanced, it does not write any data
                writer.jsonValue("");

                writer.raw("\"");

                for (String part : parts) {
                    writer.raw(part);
                }

                writer.raw("\"");

                writer.endObject();
            }
        } catch (Exception e) {
            System.err.println("\n\n");
            e.printStackTrace();
        }
    }

    public static class MyJsonWriter extends JsonWriter {
        private Writer out;

        public MyJsonWriter(Writer out) {
            super(out);
            this.out = out;
        }

        public MyJsonWriter raw(String raw) throws IOException {
            this.out.write(raw);
            return this;
        }
    }

I'm hoping for an already built-in method to do this kind of behaviour

[lyubomyr-shaydariv]

It would be a nice feature to have. I had a similar issue from the memory consumption perspective, but it was a reverse operation and considered correct string parsing too, not just "read it raw": #971. I think the method raw you implemented might be reimplemented by adding a pair of methods like begin***Literal and end***Literal. Say, something like

jsonWriter.beginStringLiteral(); // not sure for beginNumberLiteral since JSON allows arbitrary length numbers, but how to use value(Number/String) then?
while ( todo() ) {
    jsonWriter.value(nextStringChunk()); // this gets properly escaped
}
jsonWriter.endStringLiteral();

[KhunFirst]

d