search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
14.76k stars 3.79k forks source link

[source-opsgenie] Incremental approach not working for Alerts #39545

Open anamargaridarl opened 2 weeks ago

anamargaridarl commented 2 weeks ago

Connector Name

source-opsgenie

Connector Version

0.3.6

What step the error happened?

During the sync

Relevant information

The incremental sync for Opsgenie alerts is done using the query parameter: {query: updatedAt>=last updatedAt}. However, the low code connector does not specify this. Furthermore, I'm unsure if it is currently possible to define the "updatedAt>=" expression for a query parameter using the low-code approach.

Relevant log output

No response

Contribute

marcosmarxm commented 2 weeks ago

Not 100% certain about it but it is probably possible to update the requester to make something:

query: "{query: updatedAt>= {{ stream_state.get('updatedAt') }} }" if stream_state.get('updatedAt') else ""
marcosmarxm commented 2 weeks ago

The auth0 has a similar logic https://github.com/airbytehq/airbyte/blob/9458fd22fbf7c12ce8702957fe0729b1172d33cf/airbyte-integrations/connectors/source-auth0/source_auth0/manifest.yaml#L91

@anamargaridarl let me know if you want to work on this issue I can help testing and validate the change.

anamargaridarl commented 2 weeks ago

I managed to fix the issue! However, I recreated the connector alert stream on a more updated version of the yaml which does not entirely match this one - using the low code in Airbyte version 0.57.1

To resolve this, should I update the YAML connector to a more recent version and add the necessary changes?