snowplow / snowplow-badrows

Apache License 2.0
7 stars 2 forks source link

Consider generic error as a fallback option for parsing bad rows #49

Open chuwy opened 4 years ago

chuwy commented 4 years ago

We can make GenericError a some kind of "base class" for all bad rows, so if producer (enrich) produces new kind of bad rows that consumer (loader) cannot recognize yet - instead of failing it could attempt to parse it as GenericError.

28 #47

benjben commented 4 years ago

We need to be careful with this, as this can hide a parsing/incompatibility problem that needs to be addressed.

benjben commented 4 years ago

If we implement this I think that we should add a boolean to the parsing function to tell if we want to use a fallback or not.

chuwy commented 4 years ago

Both are very good points.

Another downside is that we'll be limited in structure flexibility for all bad rows. Also (or "on the other hand"), I think consumer should always be explicit about what bad rows it needs.