gojekfarm / beast

[Deprecated] Load data from Kafka to any data warehouse. BQ sink is being supported in Firehose now. https://github.com/odpf/firehose
Apache License 2.0
148 stars 23 forks source link

Messages outside of streaming insert range are not sent to DLQ #55

Open mauliksoneji opened 4 years ago

mauliksoneji commented 4 years ago

The current check for BQInsertErrors in the case where messages are out of streaming insert range relies on the response message from Bigquery API.

Since the Bigquery streaming insert ranges have changed as mentioned here, the messages are treated as failures and not OutOfBound messages. This is the reason why messages are not sent to DLQ even when they fall outside the streaming insert range.