search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.45k stars 3.98k forks source link

[source-klaviyo]: connector missing fields ahead of Klaviyo API v1 Deprecation on June 30th #39554

Open glevineLeap opened 2 months ago

glevineLeap commented 2 months ago

Connector Name

source-klaviyo

Connector Version

2.7.0

What step the error happened?

None

Relevant information

The v1/v2 legacy APIs are scheduled to be retired by Klaviyo on June 30th, 2024. As such, all source versions below 2.0 will stop working at that time and anyone using an earlier version will be forced to migrate onto a 2.0+ version.

Currently, there is still not parity in the data available between the two major versions. For example, this slack thread calls out that "the new API doesn't return profile data or profile ID with event's stream data" and this requires calling another endpoint that Airbyte doesn't currently include.

Happy to contribute if someone can point me in the right direction.

Relevant log output

No response

Contribute

marcosmarxm commented 2 months ago

@glevineLeap the API revision the Klaviyo connector uses today is 2023-10-15 which has plans to be deprecated only in 2025.

https://github.com/airbytehq/airbyte/blob/23fd80aa2a3a6f8834192e6d9a018f2fec95fc49/airbyte-integrations/connectors/source-klaviyo/source_klaviyo/streams.py#L27

marcosmarxm commented 2 months ago

@glevineLeap it is confusing to me what is the goal of this issue. Are some field missing considering the current version, can you point the Klaviyo documentation to the endpoint and fields?

glevineLeap commented 2 months ago

@glevineLeap the API revision the Klaviyo connector uses today is 2023-10-15 which has plans to be deprecated only in 2025.

@marcosmarxm Correct, but the current version is still not at data parity with the 1.x versions of the connector. So, this version of the connector won't be deprecated but the old ones that I think many people are still using will be.

@glevineLeap it is confusing to me what is the goal of this issue. Are some field missing considering the current version, can you point the Klaviyo documentation to the endpoint and fields?

Specifically, in our case we don't have a way under the new structure to link events with flows/campaigns. There's an included field in Get Events that's not currently in the Airbyte schema.

glevineLeap commented 2 months ago

There are other relevant fields (for example Campaigns no longer includes the number of recipients and theres a separate Get Campaign Recipient Estimation endpoint), but the event-flow/campaign tie is the only thing we don't have a workaround for right now.