AbsaOSS / spark-commons

Apache License 2.0
7 stars 0 forks source link

Create a Spike for error handling #83

Closed benedeki closed 1 year ago

benedeki commented 1 year ago

Background

Many of our Spark extending libraries need a way to report errors that emerged during data processing. They can be either re-implementing the preferred method of the application their main usage is expected in, or having a possible different way of error reporting than the application or simply ignore the error.

This Spike is to investigate, if it would be possible to create a generic trait to report the errors. This traits different implementation then would be provided to the library upon initialization by the higher level app therefore decoupling the style of error handling from the library that needs it.

Questions To Answer

  1. Can such a generic trait be effectively created
  2. Would that give enough flexibility and anchor-points at the same time

Desired Outcome

Have a main trait and system of supporting classes for general error handling.

### Tasks
- [x] Implement the main `trait ErrorHandling` and necessary support classes to provide general error handling
- [x] Refactor `ErrorMessage` class
- [x] Implement error handling by putting the info into column of `ErrorMessage` array
- [ ] https://github.com/AbsaOSS/spark-commons/issues/88
- [ ] https://github.com/AbsaOSS/spark-commons/issues/89
- [ ] https://github.com/AbsaOSS/spark-commons/issues/91
- [ ] https://github.com/AbsaOSS/spark-commons/issues/101
- [ ] https://github.com/AbsaOSS/spark-commons/issues/93
- [ ] #111 

Depends on #82

benedeki commented 1 year ago

This item depends on: