search

airbytehq / quickstarts

180 stars 45 forks source link

Web Analytics Stack #15

Open ThaliaBarrera opened 1 year ago

ThaliaBarrera commented 1 year ago

Integrating Google Analytics and Google Search Console.

Extract web traffic and search query data from Google Analytics and Google Search Console, transform it using dbt for insights into user behavior and search performance, and store in a data warehouse like BigQuery.

How to get started:

1cadumagalhaes commented 1 year ago

Hey, I would like to work on this one

ThaliaBarrera commented 1 year ago

Thanks @1cadumagalhaes, I have assigned it to you!

ThaliaBarrera commented 11 months ago

Hi @1cadumagalhaes, this message is just to remind you that according to the Hackathon rules you have 5 days to submit progress on the issue. I’ll give you a couple more days to submit a PR, otherwise I’ll have to mark the issue as “To be Claimed” or reassign it.

Thank you for understanding! Let me know if you have any questions.

1cadumagalhaes commented 11 months ago

Hey @ThaliaBarrera , thanks I'll submit them today, I got a little lost with the deadline but it's okay, thanks for the extra days

ThaliaBarrera commented 11 months ago

Thanks for the heads up @1cadumagalhaes! If you have questions, don’t hesitate to ask.

1cadumagalhaes commented 11 months ago

I'm finishing this one, but I got into an issue with the Google Analytics resource on Terraform (https://github.com/airbytehq/terraform-provider-airbyte/issues/42) and I think I'll have to leave only the UI configuration for this source

1cadumagalhaes commented 11 months ago

Also I had some troubles making this source work at all.

One thing I noticed is that the documentation of the source in the UI points to the wrong API on the Google Cloud API console, it sends to Analytics Reporting and Analytics API but it should lead to Google Analytics Data API.

This last one is the only necessary, and without it the connection doesn't work

1cadumagalhaes commented 11 months ago

There are some limitations with Google Search Console as well, at least for a quickstart. For some reason it isn't possible to use a Service Account to use the API without having a Google Workspace account, which isn't free (and I don't have one to test). So the only option is to use an access token with client id and etc, but since Airbyte doesn't provide a native way of requesting user permission this means I would need to develop an application to:

  1. Use the client id and secret to request user permission using a consent screen (which isn't good to use in a local environment but there must be a way)
  2. Get the access and refresh token

And then with these 4 values proceed to set up the source and connection

ThaliaBarrera commented 11 months ago

Thanks for pointing all these issues out @1cadumagalhaes! I’ll send these concerns to the team. And sorry about the time you spent trying to make this work.

In the meantime, would you like to work on another quickstart? You can choose from the ones available. If you have a stack you’re familiar with I can also create an issue for you to work on.