datahub-project / datahub

The Metadata Platform for your Data and AI Stack
https://datahubproject.io
Apache License 2.0
9.93k stars 2.94k forks source link

MCE Consumer swallows errors about misconfigured MCE events #2275

Closed frsann closed 3 years ago

frsann commented 3 years ago

Describe the bug Sending a "bad" MCE to the MCE consumer doesn't show up in the logs in a clear way. For example, sending a message where the key is avro serialized we see no indication of any problems in the MCE Consumer logs. After enabling debug logs, we see logging like:

13:44:20.181 [mce-consumer-job-client-0-C-1] DEBUG o.s.k.l.KafkaMessageListenerContainer$ListenerConsumer - Received: 1 records
13:44:20.181 [mce-consumer-job-client-0-C-1] DEBUG o.s.k.l.a.RecordMessagingMessageListenerAdapter - Processing [GenericMessage [payload={"auditHeader": null, "proposedSnapshot": <REMOVED_STUFF_HERE>}]}, "proposedDelta": null}, headers={kafka_offset=11314, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@621b2105, kafka_timestampType=CREATE_TIME, kafka_receivedMessageKey=�urn:li:dataset:(urn:li:dataPlatform:snowflake,foobar,PROD), kafka_receivedPartitionId=0, kafka_receivedTopic=MetadataChangeEvent_v4, kafka_receivedTimestamp=1616420660132}]]

which might indicate that the message never actually arrives at the MetadataChangeEventsProcessor.consume function, but is swallowed upstream by the utilized kafka adapter.

To Reproduce

  1. Send a MCE where the key is Avro-serialized.
  2. Check the MCE Consumer logs.

Expected behavior An error is logged, preferably with a clear explanation of the failure.

Screenshots

Desktop (please complete the following information):

Additional context

frsann commented 3 years ago

Stupid user error