The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Running a full sync of OpsGenie alerts seems to complete without any errors, however when looking at the resulting data in the source, alerts cut off in August 2022 with no further alerts included in subsequent syncs.
Looking at the sync logs, there is nothing that stands out, except a lot of throttling-warnings.
Manually using the OpsGenie API via curl, I can verify that there are alert records available past August 2022.
If this is a limitation in the cursor-based pagination, would it be possible to have the sync fetch alerts in descending order instead of ascending?
Relevant log output
2024-03-06 13:14:30 source > Marking stream alerts as STARTED
2024-03-06 13:14:30 source > Syncing stream: alerts
2024-03-06 13:14:30 source > Setting state of SourceOpsgenie stream to {'updatedAt': '1709644457'}
2024-03-06 13:14:30 source > Marking stream alerts as RUNNING
2024-03-06 13:14:35 platform > Records read: 375000 (99 MB)
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing alert_logs: 16081 records (24 MB)
2024-03-06 13:14:36 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 16081
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for alert_logs
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing users: 13 records (24 MB)
2024-03-06 13:14:36 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 13
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for users
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing integrations: 23 records (24 MB)
2024-03-06 13:14:36 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 23
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for integrations
2024-03-06 13:14:36 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing teams: 7 records (24 MB)
2024-03-06 13:14:36 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 7
2024-03-06 13:14:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for teams
2024-03-06 13:14:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing alerts: 2061 records (24 MB)
2024-03-06 13:14:37 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 2061
2024-03-06 13:14:37 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for alerts
2024-03-06 13:14:49 platform > Records read: 380000 (103 MB)
2024-03-06 13:14:56 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing alerts: 7524 records (24 MB)
2024-03-06 13:14:56 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 7524
2024-03-06 13:14:57 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for alerts
2024-03-06 13:15:02 platform > Records read: 385000 (107 MB)
2024-03-06 13:15:16 platform > Records read: 390000 (111 MB)
2024-03-06 13:15:17 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):85 Flushing alerts: 7604 records (24 MB)
2024-03-06 13:15:17 destination > INFO i.a.i.d.c.ClickhouseSqlOperations(insertRecordsInternal):67 actual size of batch: 7604
2024-03-06 13:15:18 destination > INFO i.a.i.d.r.InMemoryRecordBufferingStrategy(flushAllBuffers):91 Flushing completed for alerts
2024-03-06 13:15:24 source > Read 20000 records from alerts stream
2024-03-06 13:15:24 source > Marking stream alerts as STOPPED
2024-03-06 13:15:24 source > Finished syncing alerts
dipth
commented
4 months ago
Connector Name
source-opsgenie
Connector Version
0.3.1
What step the error happened?
During the sync
Relevant information
Running a full sync of OpsGenie alerts seems to complete without any errors, however when looking at the resulting data in the source, alerts cut off in August 2022 with no further alerts included in subsequent syncs. Looking at the sync logs, there is nothing that stands out, except a lot of throttling-warnings. Manually using the OpsGenie API via curl, I can verify that there are alert records available past August 2022.
If this is a limitation in the cursor-based pagination, would it be possible to have the sync fetch alerts in descending order instead of ascending?
Relevant log output
Contribute