search

dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.38k stars 154 forks source link

skips tables without jobs when merging delta tables #1803

Closed rudolfix closed 2 weeks ago

rudolfix commented 2 weeks ago

Description

  1. it was assumed that all tables in table chain will have a job. that is not true for nested tables where child tables have no rows
  2. schema evolution: presence is checked via names, not full schema comparison
  3. "heavy" object are explicitly deleted. not sure if that helps a lot... internally delta does a lot of unnecessary conversions on the arrow objects
netlify[bot] commented 2 weeks ago

Deploy Preview for dlt-hub-docs canceled.

Name Link
Latest commit 03b61ea799fbc1ff7f55b6ccbdf1461fadea3f8b
Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66e2e2a42ee8f90008dc0c4c