search

thbar / kiba

Data processing & ETL framework for Ruby
https://www.kiba-etl.org
Other
1.75k stars 87 forks source link

Error handling, logging, and alerting? #28

Closed timuckun closed 7 years ago

timuckun commented 8 years ago

You should add other blocks like ensure, on_error, and flags such as stop_on_error, log etc

Sometimes it's OK to skip one bad row and let the rest of the file process other times not so much. Also most of the ETL tends to be run unattended to there should be hooks for email alerts or sending data to a monitor.

Thanks.

thbar commented 8 years ago

Thanks for your input! Quick reply (but I'll elaborate): skipping one bad row is already possible with a bit of care, and setting up logging & alerting is also doable (I'm doing this in production on a number of deployments).

I'll prepare readme sections for this and will update this ticket once it's ready. Thanks!

thbar commented 8 years ago

(to clarify, just in case: most of my ETL deployments run unattended, indeed, and with error handling / logging enabled).

jamesohare84 commented 8 years ago

Hi @thbar

That would be an excellent addition. We are currently looking at adding error_handling and logging as part of our implementation.

Look forward to the update

thbar commented 8 years ago

@jamesohare84 thanks for the note - I'll try to deliver an article not too late :smile:

thbar commented 8 years ago

Please track issue #29 - I'm going to ship a cookbook with more concrete advice on these topics.

thbar commented 7 years ago

Follow-up: I've started sharing commonly used helpers (including the one I use for logging on production setup) in the new kiba-common gem.

I'll share back more useful bits over the coming months (either in kiba-common or as wiki pages).

thbar commented 7 years ago

Closing for now - I will craft an article / kb at some point with more details about this.