dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.38k stars 154 forks source link

abort load package and raise exception on terminal errors in jobs #1749

Closed rudolfix closed 2 weeks ago

rudolfix commented 1 month ago

Background by default dlt will move terminally failed jobs into failed_jobs folder and will not raise any exceptions. users can change load.raise_on_failed_jobs=true config option to abort the package and raise exception afterwards.

this PR makes this behavior default.

background: https://dlthub.com/docs/running-in-production/running#handle-exceptions-failed-jobs-and-retry-the-pipeline

Requirements PR 1:

  1. Switch raise_on_failed_jobs to true
  2. Update run in production docs
  3. Update failing tests. There may be a lot of them. Add unit tests to test_dummy_client (I hope we test this config flag somewhere in it)

PR 2:

  1. Add a new cli/pipeline method to retry aborted package (moves failed jobs back to new). extend this command: https://dlthub.com/docs/reference/command-line-interface#get-the-load-package-information and add custom parser with "retry-aborted". which:
    • will remove "aborted" flag and move package from "completed" to "normalized"
    • move all failed jobs back to new jobs and delete failed messages.
willi-mueller commented 2 weeks ago

@rudolfix The only downside of this change might be that the default strategy for the write_disposition = "replace" would truncate the table and not add any data because the load package failed.

Could we change the default strategy to ' insert-from-staging ' to minimize the chances of users deleting their data on error?

Background: https://dlthub.com/docs/general-usage/full-loading#the-truncate-and-insert-strategy