dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.65k stars 176 forks source link

save collected load job metrics #1744

Open rudolfix opened 2 months ago

rudolfix commented 2 months ago

Background We should store metrics when they go into terminal state. Currently if package is partially loaded and the process is killed, we lose metrics of completed jobs.

Proposed implementation: store them in a separate file along the job file - the same way we store failed messages.