The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Tell us about the new integration you’d like to have
Which source and which destination? Which frequency?
Instagram insights to Google BigQuery, every <10 minutes(!). It's actually from the Facebook Graph API so it might be possible to reuse some aspects of the Facebook Marketing Source Connector.
Describe the context around this new integration
Which team in your company wants this integration, what for? This helps us understand the use case.
I have already built this as a custom extractor for a client using Google Cloud Functions - the reason is that:
They have multiple Instagram accounts tied to a single business account (currently deployed on <5 but in future it might be 100+)
There are some metrics which are only given as total 'lifetime' but for which it is extremely valuable to compute the change in metrics over time (hence polling the API every 10 minutes)
Image URLs change over time so it's necessary to refresh metadata on a regular basis
Story insights disappear after 24 hours so it's important to poll and store results at short intervals
Some metrics are contained in the response from the media endpoint (https://graph.facebook.com/{graph-api-version}/{ig-user-id}/media?fields={fields}&access_token={access-token}), some story-related ones from the story endpoint (https://graph.facebook.com/{graph-api-version}/{ig-user-id}/stories?fields={fields}&access_token={access-token}) and more detailed insights from the insights endpoint (https://graph.facebook.com/v9.0/{ig-media-id}/insights?metric={metrics}&access_token={access-token}). Additionally account insights are returned from the user insights endpoint (https://graph.facebook.com/v9.0/{ig-user-id}/insights?metric={metric}&period={period}&since={since}&until={until}&access_token={access-token})
Describe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
It was not viable to do this via e.g. Rivery as the sheer number of data transfers would have been prohibitively expensive, and the multi-account to single business account makes it incompatible with other tools. It feels like a great use-case for us to try out Airbyte, and I would be happy to share the (Python) code as deployed.
I am already running it via a set of Cloud Function (orchestrated via PubSub from other Cloud Function so each invocation only queries data from a single account), however there is an ongoing cost associated with this and I would rather move it to a dedicated platform and monitor/manage it alongside other flows instead of as custom code.
Tell us about the new integration you’d like to have
Which source and which destination? Which frequency?
Instagram insights to Google BigQuery, every <10 minutes(!). It's actually from the Facebook Graph API so it might be possible to reuse some aspects of the Facebook Marketing Source Connector.
Describe the context around this new integration
Which team in your company wants this integration, what for? This helps us understand the use case.
I have already built this as a custom extractor for a client using Google Cloud Functions - the reason is that:
https://graph.facebook.com/{graph-api-version}/{ig-user-id}/media?fields={fields}&access_token={access-token}), some story-related ones from the story endpoint (https://graph.facebook.com/{graph-api-version}/{ig-user-id}/stories?fields={fields}&access_token={access-token}) and more detailed insights from the insights endpoint (https://graph.facebook.com/v9.0/{ig-media-id}/insights?metric={metrics}&access_token={access-token}). Additionally account insights are returned from the user insights endpoint (https://graph.facebook.com/v9.0/{ig-user-id}/insights?metric={metric}&period={period}&since={since}&until={until}&access_token={access-token})Describe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
It was not viable to do this via e.g. Rivery as the sheer number of data transfers would have been prohibitively expensive, and the multi-account to single business account makes it incompatible with other tools. It feels like a great use-case for us to try out Airbyte, and I would be happy to share the (Python) code as deployed.
I am already running it via a set of Cloud Function (orchestrated via PubSub from other Cloud Function so each invocation only queries data from a single account), however there is an ongoing cost associated with this and I would rather move it to a dedicated platform and monitor/manage it alongside other flows instead of as custom code.