confluentinc / confluent-kafka-javascript

Confluent's Apache Kafka JavaScript client
https://www.npmjs.com/package/@confluentinc/kafka-javascript
MIT License
64 stars 5 forks source link

Headers (outbound and inbound) do not comply with KafkaJS type definitions for eachBatch consumers #129

Open EricMCornelius opened 3 days ago

EricMCornelius commented 3 days ago
os: ubuntu 24.04 ARM
node_version: 22.9.0
npm_version: 10.9.0
confluent-kafka-javascript version: v0.2.1

Steps to Reproduce

Any kafkajs consumer using an eachBatch configuration has invalid headers structure. Console logging from a basic setup when sending headers yields:

{
  key: null,
  value: <Buffer 7b 22 74 79 70 65 22 3a 22 74 65 73 74 5f 65 76 65 6e 74 22 2c 22 65 76 65 6e 74 22 3a 7b 22 6d 65 73 73 61 67 65 22 3a 22 65 76 65 6e 74 20 23 31 34 ... 271 more bytes>,
  timestamp: '1729305041677',
  attributes: 0,
  offset: '1785544',
  size: 321,
  leaderEpoch: 20,
  headers: {
    '0': {
      org: <Buffer 37 33 38 30 33 35 64 63 2d 30 34 34 38 2d 35 66 38 64 2d 61 66 30 33 2d 35 30 30 32 36 61 61 30 38 36 36 62>
    }
  }
}

Additional context This was fixed for eachMessage but not for eachBatch.

milindl commented 1 day ago

Thanks @apeloquin-agilysys and @EricMCornelius , I'll be taking a look.