lisad / phaser

The missing layer for complex data batch integration pipelines
MIT License
4 stars 1 forks source link

Refactors Context, especially to index and store evens differently #104

Closed lisad closed 5 months ago

lisad commented 5 months ago

One goal was to separate errors, warnings and dropped-row events by phase

While doing this, it became necessary to keep track of current phase (although sometimes we can pass the phase into an event creation, sometimes we can't - steps don't know what phase they're in and can't include that in adding errors/warnings.

Because of messing with the signatures of handling exceptions, it became easy just to remove the error_policy from phase and move it to context like we discussed last time.

With this change, Context is also hiding the way it organizes errors/warnings from outside objects. Instead of addressing its dicts directly, we have questions like "phase_has_errors" and "row_has_errors" and the getter for events "get_events". We probably need a bit more work in this direction, but this PR takes the hit of changing all the tests that were directly querying the context's structures for storing events.

I think this is better on the whole though it does point to some further improvements. I'd like to refactor the exception handling in Context now that it has changed in shape a bit