Make base64 encoding optional

[johanvanderkuijl]

We’re using Logstash to write event data to a Kafka queue. The consumer of this queue expects an Avro binary blob (or fragment, the terminology seems to differ sometimes). It seems however, that using the Avro codec provided by Logstash for the Kafka output adapter, results in a base64 encoded version of the binary fragment being produced to Kafka. This is also clear from the source code:

https://github.com/logstash-plugins/logstash-codec-avro/blob/e81dadd46f54cc88478ff843cef9bbf8ce9753e2/lib/logstash/codecs/avro.rb#L93

Do you agree that this encoding should be configurable, like 'base64encoding => true/false' ? What is it's use anyway? Should I make a PR for this issue?

Thanks in advance, Johan

[original-brownbear]

@Rubyan yea this makes a lot of sense to me. The reason we're doing the base64 encoding on top of Avro here is that there were some decoding issues when handling raw byte[] on the Ruby side of things in the Avro deserialization (when used together with Kafka String encoder that coerced data to UTF8).

If you wish to provide a PR that makes this optional (let's default to true though to not break the round-trip use cases we initially fixed via the base64 round) I'm happy to review + merge this. Otherwise, I can also look into fixing this mid next week.

[guillaume7]

Hi, we are facing same issue than @Rubyan.

Do you have any intention to make base64 encoding optional?

Thank you and best regards, Guillaume

[lucabelluccini]

The mandatory base64 wrapping prevents to decode only specific fields of the Avro payloads (e.g. decode only the fields specified in the schema, which might be less than the ones present in the writer schema).

[th0ger]

@original-brownbear Is this fixed?