matf/stop-overwriting-state-on-empty-state

z3z1ma / target-bigquery

target-bigquery is a Singer target for BigQuery. It supports storage write, GCS, streaming, and batch load methods. Built with the Meltano SDK.

MIT License

27 stars 34 forks source link

Closed shopmatfournier closed 1 year ago

shopmatfournier commented 1 year ago

closes #52

if the state message is empty (that is, no state message was processed) then do not attempt to write the state as this can overwrite state in the event a tap produces no records (or no state message)

firehist commented 1 year ago

Hello there,

Thanks for the job. I spotted the same behavior, and I use the latest released version with commit sha 3952428bb043d1db80250e1f11cbcad9c8b18ecd

When it runs without data coming from extractors, it write into the stage empty stage, which make a full run next time.

Do you guys have any suggestions?

Also in Meltano slack : https://meltano.slack.com/archives/C01TCRBBJD7/p1689942408136719

aminebeh commented 1 year ago

Hi, I have encoutered the same behavior as @firehist.

Just wanted to know if it's intended behavior (that should be handled with the right config) or an oversight ?

Thanks in advance for your work 🙏