mtth / avsc

Avro for JavaScript :zap:
MIT License
1.27k stars 147 forks source link

Unable to deserialize message value buffer from Kafka #389

Closed jcjp closed 2 years ago

jcjp commented 2 years ago

Why is the decoding process failing when the message is coming from Kafka but when its a local variable from the code everything works fine here is some of my code snippet:

let buffer = type.toBuffer(event)
let abuffer = atype.toBuffer(event)

// At this point all of them are Buffer values
console.log({
  kafka: message.value,
  local: buffer,
  abuffer
})

let obj = type.fromBuffer(buffer)
let aobj = atype.fromBuffer(abuffer)
// The line below is where the error happens when I try to decode Kafka message value buffer
let kobj = atype.fromBuffer(message.value)
console.log({
  local: obj,
  aobj,
  kobj
})

167291279-e0647b38-9a64-4077-9d5f-6d1a90430e7c~2

I am producing a message from a docker Kafka image console. Here is a sample payload:

{
  event: 'EVENT'
  payload: {
    name: 'NAME',
    desceription: 'OPTIONAL'
  }
  time: 0,
  version: 0
}

It might be a mismatch with the schema because something ia messing with the message, however looking at the logs (screenshot) the payload is what I expected. Is it safe to just slice the buffer I tried slicing it with a value of 9 but still same error.

I could not use fromString as well and toBuffer.

jcjp commented 2 years ago

I have found the fix for the issue, it seems there is a mismatch with the schema as Kafka needs double quotes for JSON object keys when passed. More information on this link: Json parse on message value string not working

Apparently I only got this working locally, when using another application to post a message through HTTP request and encoding the data via AVRO I get the same error when decoding on my consumer side.