feature request - option to respond as ndjson

[arikastarvo]

It would be great if there were option to respond as ndjson instead of plain json. This would allow to process the data with tools that handle events line by line.

So instead of responding: [{"_id":1,"data":"foo"},{"_id":2,"data":"bar"}] .. the responsw would be:

{"_id":1,"data":"foo"}
{"_id":2,"data":"bar"}

[ujibang]

Hi @arikastarvo, this can be easily achieved with a simple response Interceptor.

This works quite well:

package org.restheart.mongodb.interceptors;

import org.restheart.exchange.MongoRequest;
import org.restheart.exchange.MongoResponse;
import org.restheart.plugins.InterceptPoint;
import org.restheart.plugins.MongoInterceptor;
import org.restheart.plugins.RegisterPlugin;
import org.restheart.utils.BsonUtils;

/**
 * transforms the response to ndjson
 *
 * @author Andrea Di Cesare {@literal <andrea@softinstigate.com>}
 */
@RegisterPlugin(name = "ndJsonTransformer",
        description = "used ndjson for GET /coll if query string contains the 'ndjson'",
        interceptPoint = InterceptPoint.RESPONSE)
public class NdJsonTransformer implements MongoInterceptor {
    @Override
    public void handle(MongoRequest request, MongoResponse response) {
        var ndjson = new StringBuilder();

        response.getContent().asArray().stream()
            .forEachOrdered(doc -> ndjson.append(BsonUtils.toJson(doc)).append("\n"));

        response.setContentType("application/x-ndjson");
        // cannot use response.setContent() since MongoResponse handles content as BosnValue
        // we need to use a custom sender
        response.setCustomerSender(() -> response.getExchange().getResponseSender().send(ndjson.toString()));
    }

    @Override
    public boolean resolve(MongoRequest request, MongoResponse response) {
        return request.isHandledBy("mongo")
                && request.getQueryParameters().containsKey("ndjson")
                && !request.isInError()
                && request.isCollection()
                && request.isGet()
                && response.getContent() != null
                && response.getContent().isArray();
    }
}

example:

$ http -b -a admin:secret :8080/coll\?ndjson
{"_id":{"$oid":"60a4e96420e94366b9a17a2e"},"a":1,"timestamp":{"$date":1621420388152},"_etag":{"$oid":"60a4e96420e94366b9a17a2c"}}
{"_id":{"$oid":"60a4e936b829d77e87bf967a"},"a":1,"timestamp":{"$date":1621420342981},"_etag":{"$oid":"60a4e936b829d77e87bf9678"}}
{"_id":{"$oid":"60a4e88b7144381ae30070c4"},"a":1,"timestamp":{"$date":1621420171279},"_etag":{"$oid":"60a4e88b7144381ae30070c2"}}
{"_id":{"$oid":"60a4e83898c50c5d88267a5c"},"a":1,"timestamp":{"timestamp":{"$date":1621420088325}},"_etag":{"$oid":"60a4e83898c50c5d88267a5a"}}
{"_id":{"$oid":"60a4e75b6ec2af4222313c97"},"a":1,"_etag":{"$oid":"60a4e75b6ec2af4222313c95"}}
{"_id":{"$oid":"60a4e61e4fda423197e16613"},"a":1,"_etag":{"$oid":"60a4e61e4fda423197e16611"}}
{"_id":{"$oid":"609418d67bfd2323ef0bcb25"},"a":"instead of ö and ü","_etag":{"$oid":"609418d67bfd2323ef0bcb24"}}
{"_id":{"$oid":"6094188a7bfd2323ef0bcb23"},"a":"é@ò ààààà","_etag":{"$oid":"6094188a7bfd2323ef0bcb22"}}
{"_id":{"$oid":"609418787bfd2323ef0bcb21"},"a":1,"_etag":{"$oid":"609418787bfd2323ef0bcb1f"}}
{"_id":{"$oid":"608c83361edd1f68824d2640"},"a":1,"_etag":{"$oid":"608c83361edd1f68824d263f"}}
{"_id":{"$oid":"608c83341edd1f68824d263e"},"a":1,"_etag":{"$oid":"608c83341edd1f68824d263c"}}
{"_id":"a","array":[{"a":1,"date":{"$date":1616428047474}},{"a":2,"date":{"$date":1616428047474}}],"_etag":{"$oid":"6058bc0fa2221b1a0e65aea6"},"d":{"$date":1616427774956}}

[arikastarvo]

Great, thanks! I tried the same approach at first and almost got it but I didn't see the setCustomerSender() option. I did have to change the BsonUtils class to JsonUtils to work (at least for the commons-5.3.5 package in maven), otherwise it works out-of-the-box.