search

snowplow / enrich

Snowplow Enrichment jobs and library
https://snowplowanalytics.com
Other
21 stars 38 forks source link

Switch bad row type from EnrichmentFailures to SchemaViolations for s… #883

Closed benjben closed 6 months ago

benjben commented 7 months ago

There are 3 places that currently produce an EnrichmentFailures bad row whereas the more appropriate type is SchemaViolations:

  1. When the input fields of the HTTP requests are mapped to the atomic event.
  2. When the enrichments contexts get validated.
  3. When the atomic fields lengths get validated.

For 1 and 3, the error should be mapped into an Iglu ValidationError with the atomic schema referenced.

Before this change, if there was any error in the mapping of the atomic fields, a bad row would get emitted right away and we would not try to validate the entities and unstruct event. Now all these errors are wrapped inside a same SchemaViolations bad row.

Likewise, before this change when an enrichment context was invalid, we were emitting a bad row right away and not checking the lengths of the atomic fields. Now all these errors are wrapped inside a same SchemaViolations bad row.

benjben commented 6 months ago

Hey @istreeter , I addressed all your feedback, please have a look