Support for other formats like Smile or CBOR

[albertonavarro]

As a tormented developer trying to improve logging in my company I realised that ElasticSearch supports Smile and CBOR as input format.

I wonder if it is in your plans to allow output in these formats

So there could be an improvement in disk space, network usage and cpu usage by using binary formats instead of pure JSON.

If it's not and you guys think it could be a good idea, and you want to lead me to the right classes, I might give it a try.

[philsttr]

Fortunately, jackson makes it easy to support additional formats by providing implementations of JsonFactory that write in those other formats. For example SmileFactory and CBORFactory are subclasses of JsonFactory

Unfortunately, logstash-logback-encoder currently depends on MappingJsonFactory instead of just the base JsonFactory, and MappingJsonFactory is JSON specific.

The easiest way to support other formats would be to have a JsonFactoryDecorator that returned an implementation of JsonFactory that writes in the desired format. However, since JsonFactoryDecorator currently operates on a MappingJsonFactory, this cannot be done without making a backwards incompatible change to JsonFactoryDecorator.

I just pushed a branch named dataformats that makes this backwards incompatible change, and I was able to write in Smile and CBOR using a JsonFactoryDecorator that returned the respective factories. For example:

import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.smile.SmileFactory;

public class SmileJsonFactoryDecorator implements JsonFactoryDecorator {

    @Override
    public JsonFactory decorate(JsonFactory factory) {
        SmileFactory smileFactory = new SmileFactory();
        ObjectMapper mapper = new ObjectMapper(smileFactory);
        smileFactory.setCodec(mapper);
        return smileFactory;
    }
}

Since I just released 5.0 with several backwards incompatible changes, I'm hesitant to release another backwards incompatible version so quickly.

I'll need to see if there is an alternative option that is backwards compatible.

[albertonavarro]

Awesome, thanks, let us know if there is anything we can do to help.

[philsttr]

I was able to make this somewhat backwards compatible in the dataformats branch by providing default methods on the JsonFactoryDecorator interface. I had to bump the minimum java version to 1.8 though. So still will need a major version bump.

[philsttr]

Available in 6.0