search

confluentinc / kafka-rest

Confluent REST Proxy for Kafka
https://docs.confluent.io/current/kafka-rest/docs/index.html
Other
47 stars 647 forks source link

Kafka REST proxy mandates field with null values for optional fields of AVRO schema #427

Open keyankay opened 6 years ago

keyankay commented 6 years ago

I have plenty of optional fields in my AVRO schema, and i do not want to send value as null for all the fields that are optional (there are about 20 optional fields and 5 mandatory fields) via JSON

The Kafka REST proxy is asking for the attribute to be specified even though default has been specified to null.

Ex: "schema": " { "namespace" : "sample", "type" : "record", "name" : "samp", "fields": [ {"name" : "header", "type": { "type" : "record", "name" : "sub_header", "fields" : [ {"name" : "version", "type" : "string", "default" : "NONE"}, {"name" : "id", "type" : "string", "default" : null}, {"name" : "messageID", "type" : "string", "default" : "NONE"} ] } } ] }"}

Also i tried using union {"name" : "id", "type" : ["null","string"], "default" : null} . It did not help still

Now when i want to populate the JSON data: i want to avoid sending the field "id" in the curl command, but it is failing

curl -X POST_schema_id": 93, "records": [{"value": {"sub_header" : {"version" : "images/img.png","messageID" : "123" }}}]}' "http://localhost:8082/topics/avrotest2"

{"error_code":42203,"message":"Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Expected field name not found: id"}

However the below works: curl -X POST_schema_id": 93, "records": [{"value": {"sub_header" : {"version" : "images/img.png","id":null,"messageID" : "123" }}}]}' "http://localhost:8082/topics/avrotest2"

I see that it has been fixed in AVRO for Java script : mtth/avsc#118

yatharthranjan commented 5 years ago

Is there a particular reason this has not been attended to yet? Seeing as the major USP of avro is schema evolution, this is causing a great obstacle to that. We are facing similar problem -- https://github.com/RADAR-base/RADAR-Schemas/issues/153

Would be good to know if there is a planned fix for this in the near future @ewencp @mageshn .

ijesh commented 4 years ago

Running into a similar issue myself. How are people getting around this issue? are there any workarounds? Doesn't seem like this issue is going to get looked at anytime soon. The issue seems to have been introduced here per this comment. Open ticket AVRO-2447

reneveyj commented 4 years ago

Any workaround? This is really needed.

RaviVaranasi commented 3 years ago

+1 on this. Seems odd to define default values only to state to client that it is required

Mnantier commented 2 years ago

I'm also currently running into this issue. Has anyone figured out a solution for this ?