search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.18k stars 4.14k forks source link

[source-google-data-api] Connector breaking when using multiple property ids #31123

Open lionelmm opened 1 year ago

lionelmm commented 1 year ago

Connector Name

Google Analytics 4 (GA4)

Connector Version

1.6.0

What step the error happened?

During the sync

Relevant information

When setting up a ga4 connector to bigquery everything works fine including the sync to the destination. Screenshot 2023-10-06 at 13 52 00

However, when i try to use more than one property id Screenshot 2023-10-06 at 13 54 33

the sync breaks immediately and also doesn't let me go back to the connection to edit it etc Screenshot 2023-10-06 at 13 53 52

even when i reduce the property ids afterwards the source will not work with any connector any more.

I have tried all the property ids individually and they all work without any issues

Relevant log output

No response

Contribute

Talalqaddoumi commented 10 months ago

I had the same issue but when trying to refresh the connection to get update the source schema. The approach of creating a stream per property proves unhelpful and leads to complications when refreshing the source schema, especially after the addition of numerous properties. Currently, I'm facing difficulties generating and refreshing the source schema in a setup with 160 properties.

Here is an illustration of the challenge: image

marcosmarxm commented 10 months ago

@Talalqaddoumi can you get the API response in the Network tab to check what error happened?

Talalqaddoumi commented 10 months ago

@marcosmarxm I think this is causing an issue in get_stream_properties API, I tired calling the api separately to get the stream and getting this error: image and this is the full api response : { "statusCode": 408, "type": "https://reference.airbyte.com/reference/errors", "status": 408, "title": "try-again", "detail": "Updating cache latest source schema in progress. Please try again with cache on." }