search

singer-io / tap-chargebee

Singer.io tap for extracting data from the Chargebee API
https://www.singer.io/
GNU Affero General Public License v3.0
7 stars 32 forks source link

Repeatedly synchronizing entire Invoices table #30

Open brylie opened 4 years ago

brylie commented 4 years ago

We are using Stitch-managed version of Singer with the tap-chargebee enabled and noticed that we were hitting our row limit.

The Chargebee tap is replicating all of the rows in the Invoices table repeatedly, instead of the new rows only. What can we do to prevent the tap-chargebee from replicating the entire table, and all related tables, each night?

brylie commented 4 years ago

Specifically, does this tap support INCREMENTAL_UPDATE?

cb-nandita commented 3 years ago

@brylie, Hope you are doing well. Could you please check and confirm if there are any errors in the Stitch logs? Are all the syncs successful ?

brylie commented 3 years ago

I will re-enable the Invoices table sync.

brylie commented 3 years ago

I re-enabled the invoices table sync in Stitchdata.com and the row count skyrocketed:

image

There do not appear to be any errors in the replication log. I will check again tomorrow to see if we have the same row replication issue.

brylie commented 3 years ago

The weird thing above is it says there were 7.6 M rows loaded into the data warehouse that day, while the bar chart shows 37.5 M total rows loaded. I don't understand the discrepancy.

brylie commented 3 years ago

Here is the usage increase after re-enabling the Invoices table sync:

image

brylie commented 3 years ago

The spike in rows loaded seems to have passed, and things are back to the previous, upwards cumulative trend.

image