pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Log exceptions produced by tasks, even if `fail_fast=False` #85

Closed NMAC427 closed 1 year ago

NMAC427 commented 1 year ago

This is a potential fix for #34.

This PR would change the way exceptions get logged, such that whenever a task raises an exception, it gets logged at the ERROR level. Previously, it would just log that the task failed (at the WARNING level) without logging the specific exception.

Bildschirmfoto 2023-06-06 um 15 12 11