llofberg / kafka-connect-rest

Kafka Connect REST connector
Apache License 2.0
108 stars 55 forks source link

Json sink #2

Closed ben-manes closed 6 years ago

ben-manes commented 6 years ago

The current behavior is the use the value's toString() POST with. When using this and the JsonConverter, the value is a Map<String, Object>. This is stringified as {key1={key2=value2}} which is not desirable. Instead, I would like to serialize it to json using Kafka's Jackson dependency.

https://github.com/llofberg/kafka-connect-rest/blob/3a32506b65cbc04759fe1960d768f05a3147671e/src/main/java/com/tm/kafka/connect/rest/converter/BytesPayloadConverter.java#L25-L28

Can be changed to,

 // Convert to String for outgoing REST calls 
 public String convert(SinkRecord record) { 
    try {
      return mapper.writeValueAsString(record.value());      
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
 }

This of course may not be desirable for others. I'd recommend having a pluggable serializer, as the current behavior was surprising when integrating with a source pushing Structs.

llofberg commented 6 years ago

I added a JacksonPayloadConverter class as you suggested. It's used in the Spring example. https://github.com/llofberg/kafka-connect-rest/blob/master/examples/spring/config/sink.json#L9

Is that what you had in mind?

ben-manes commented 6 years ago

Yep, thanks!