search

allegro / json-avro-converter

JSON to Avro conversion tool designed to make migration to Avro easier.
Other
278 stars 99 forks source link

getting error tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro Path in schema: --> payload__id #82

Open gurjit-sandhu opened 2 years ago

gurjit-sandhu commented 2 years ago

here is the avro schema - { "namespace" : "com.fetch.test", "name" : "outer_record", "type" : "record", "fields" : [ { "name" : "payloadid", "type" : "string" }, { "name" : "payload_createdDate", "type" : "long" }, { "name" : "payload_lastUpdate", "type" : "long" }, { "name" : "payload_firstName", "type" : "string" }, { "name" : "payload_lastName", "type" : "string" }, { "name" : "payload_email", "type" : "string" }, { "name" : "payload_birthDate", "type" : "long" }, { "name" : "payload_phoneNumber", "type" : "string" }, { "name" : "payload_last4DigitsOfPhoneNumber", "type" : "string" }, { "name" : "payload_signUpPlatform", "type" : "string" }, { "name" : "payload_bonusPointScheduleId", "type" : "string" }, { "name" : "payload_allowedScopes", "type" : { "type" : "array", "items" : "string" } }, { "name" : "payload_role", "type" : "string" }, { "name" : "payload_lifetimeSavingsFromCoupons", "type" : "double" }, { "name" : "payload_deactivatedReason", "type" : [ "null" ] }, { "name" : "payload_userTosAcceptance", "type" : { "type" : "array", "items" : "string" } }, { "name" : "payload_lastPointsViewed", "type" : [ "null" ] }, { "name" : "payloadmonthlyBudget", "type" : [ "null" ] }, { "name" : "payloaddeleted", "type" : "boolean" }, { "name" : "payloaddb", "type" : "string" }, { "name" : "payloadrs", "type" : "string" }, { "name" : "payloadcollection", "type" : "string" }, { "name" : "payloadop", "type" : "string" }, { "name" : "payload___ts_ms", "type" : "long" }, { "name" : "_id", "type" : "string" } ] }

getting exception -- org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_5, processor=KSTREAM-SOURCE-0000000000, topic=mongoDev.Fetch_V2_Dev.Users, partition=5, offset=188, stacktrace=tech.allegro.schema.json2avro.converter.AvroConversionException: Failed to convert JSON to Avro at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:53) at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:41) at tech.allegro.schema.json2avro.converter.JsonAvroConverter.convertToGenericDataRecord(JsonAvroConverter.java:66) at topicFlatten.flattenJSON.parseJson(flattenJSON.java:32) at topicFlatten.jsonTopicFlattenStream.lambda$flattenJSONStream$0(jsonTopicFlattenStream.java:62) at org.apache.kafka.streams.kstream.internals.AbstractStream.lambda$withKey$1(AbstractStream.java:106) at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40) at org.apache.kafka.streams.processor.internals.ProcessorAdapter.process(ProcessorAdapter.java:71) at org.apache.kafka.streams.processor.internals.ProcessorNode.lambda$process$2(ProcessorNode.java:181) at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:884) at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:181) at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:281) at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:260) at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:219) at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:86) at org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:731) at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:879) at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:731) at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1177) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:753) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:583) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:556) Caused by: Path in schema: --> payload__id at org.apache.avro.generic.GenericData.getDefaultValue(GenericData.java:1173) at org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:142) at org.apache.avro.generic.GenericRecordBuilder.getWithDefault(GenericRecordBuilder.java:250) at org.apache.avro.generic.GenericRecordBuilder.build(GenericRecordBuilder.java:227) at org.apache.avro.generic.GenericRecordBuilder.build(GenericRecordBuilder.java:32) at tech.allegro.schema.json2avro.converter.types.RecordConverter.convertValue(RecordConverter.java:37) at tech.allegro.schema.json2avro.converter.types.RecordConverter.convertValue(RecordConverter.java:14) at tech.allegro.schema.json2avro.converter.types.AvroTypeConverterWithStrictJavaTypeCheck.convert(AvroTypeConverterWithStrictJavaTypeCheck.java:22) at tech.allegro.schema.json2avro.converter.CompositeJsonToAvroReader.read(CompositeJsonToAvroReader.java:92) at tech.allegro.schema.json2avro.converter.JsonGenericRecordReader.read(JsonGenericRecordReader.java:49) ... 21 more

gurjit-sandhu commented 2 years ago

please ignore. was issue with avro version and able to resolve it