search

snowplow-incubator / snowplow-lake-loader

Snowplow Lake Loader
Other
0 stars 3 forks source link

Implement alerting and retrying mechanisms #73

Closed spenes closed 4 months ago

spenes commented 4 months ago

Jira ref: PDP-1321

Two features has been added in this PR: alerting and retrying

For alerting, webhook method is used similar to other Snowplow apps. Alert message is sent to URL given in the config. Alerts are sent for some error cases, not for all of them. In this PR, it is implemented such that it is sent only for setup errors. The error cases where alert sent can be extended in the future, of course.

For retrying, two retry policies can be defined similar to Snowflake Loader. One of them is for setup errors and other one is for transient errors. Alert would be sent only for setup errors, not for transient errors.

Also, possible setup error cases for Iceberg/Glue/S3 are added in this PR as well. Error cases for other destinations/table formats will be added later.

One other thing I want to mention is changes on the dependencies and SBT packages. Some of the dependencies are switched from runtime to compile-time. This is done to be able to use exception classes in those dependencies in the project while trying to determine whether thrown exception is for one of the setup error cases or not. Also, new iceberg package is created to be able to use Iceberg specific exception classes while checking setup errors.