Open satyanash opened 1 year ago
Incremental sync mode added to InvoiceLineItems in version 4.0.0
It seems like this issue was closed while the InvoiceLineItems stream had never been incremental. Hence, I'll re-open the issue
I'd like to call out that SubscriptionItems
is also not incremental, as of version v5.3.9, despite having a created
timestamp that could conceivably be used (or tied to its Subscription
parent)
SubscriptionItems
and InvoiceLineItems
represented 99% of our billed rows per sync, so we had to separate them into separate connections to sync on different schedules than the incremental tables.
Connector Name
source-stripe
Connector Version
3.17.4
What step the error happened?
During the sync
Revelant information
There are two streams in the Stripe Source:
invoices
invoice_line_items
which is a "sub-stream" of invoices.The
invoice_line_items
does not support the "Incremental | Append" sync mode, even though its parent streaminvoices
supports it. Any of the two remaining options cause a Full Refresh on theinvoices
stream to happen on every sync, even if that parent stream is set to "Incremental | Append" mode. I have attached the relevant log lines which indicate that Airbyte does two separate syncs, and fetching a super-set of the data when syncing theinvoice_line_items
from the parent stream. This also defeats the purpose of having "Incremental | Append" on the parent stream, since the sub-stream causes a "Full Refresh ..." of the parent stream anyway and takes a much longer time.Relevant log output
Contribute