Bulk Load CDK: Tolerate empty streams and update on empty batches

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

https://airbyte.com

Other

16.27k stars 4.15k forks source link

Bulk Load CDK: Tolerate empty streams and update on empty batches #48591

Closed johnny-schmidt closed 1 day ago

johnny-schmidt commented 3 days ago

What

I found this fixing legacy DATs but am pulling it out because it's breaking some of @frifriSF59 's attempted test fixes.

skips processing empty spill files to avoid cryptic XML AWS errors
pushes a dummy batch update to force state update anyway (@benmoriceau this is similar to your issue; we should break out a new event so this is less hacky)
treats a fully processed empty stream as ipso facto complete
tests

vercel[bot] commented 3 days ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment