Add basic extraction log parsing feeding into stream load wait logic

msnidal / dagster-stitch

Stitch ETL integration for Dagster

Apache License 2.0

0 stars 0 forks source link

Closed msnidal closed 1 year ago

msnidal commented 1 year ago

Add extraction log endpoint to resources to parse out which streams actually have new rows and will be loaded
Improve load wait logic to incorporate above - is now at least materializing assets after running through locally!
Standardize timezone to UTC and fixes time compare in polling logic
Misc. improvements and bugfixes

Fixes for at least a subset of cases #4 and solves #5!

msnidal commented 1 year ago

Also closes out #8 although the README definitely has a ways to go