frappe / insights

Open source analytics / business intelligence tool (BI)
https://frappe.io/insights
GNU Affero General Public License v3.0
437 stars 214 forks source link

adding Google BigQuery to Insights as data sources and use queries to visualize data #324

Open sambhxv opened 1 month ago

sambhxv commented 1 month ago

Hi, I was working on adding Google's BigQuery as an Insights data source but have been confused while doing so. I was expecting the entire flow to be bit modular, but turns out I would have to make a lot more changes than presumed. It was kind of obvious to make a whole lot of changes given the fact that PostgresDB / MariaDB / Frappe site-db provide username and password based connection. Connection string can also be used to establish the connection in these cases. But when it comes to BigQuery, the connection is established using a service-account based authorization method.

image

I'm having some issues primarily with the understanding of what are the steps after a connection is established. Until now, I have created a doctype called custom-data-source which takes a project ID and service-account as parameters and establishes a connection to the database.

I have made the changes to this fork. I'm looking for a quick review of the progress I've made. There is a lot of clutter to clean up, bear with me🙂

I have a few more questions:

sambhxv commented 2 days ago

follow-up, I have made some tweaks around the system and was able to list tables and fetch data from those tables using the underlying methods, had to add google-bigquery-dialect in order to make it work.

Here's the implementation for the same: Internal Insights

nextchamp-saqib commented 1 day ago

The new implementation looks good!

sambhxv commented 1 day ago

@nextchamp-saqib looks merge-able? should I raise a PR to insights/develop?