adding Google BigQuery to Insights as data sources and use queries to visualize data

[sambhxv]

Hi, I was working on adding Google's BigQuery as an Insights data source but have been confused while doing so. I was expecting the entire flow to be bit modular, but turns out I would have to make a lot more changes than presumed. It was kind of obvious to make a whole lot of changes given the fact that PostgresDB / MariaDB / Frappe site-db provide username and password based connection. Connection string can also be used to establish the connection in these cases. But when it comes to BigQuery, the connection is established using a service-account based authorization method.

I'm having some issues primarily with the understanding of what are the steps after a connection is established. Until now, I have created a doctype called custom-data-source which takes a project ID and service-account as parameters and establishes a connection to the database.

I have made the changes to this fork. I'm looking for a quick review of the progress I've made. There is a lot of clutter to clean up, bear with me🙂

I have a few more questions:

• Am I following the right path?
• What are the steps further?
• How do I display the tables of my BigQuery DB to Insights dashboard from here?
• How do query builders work?
• Are query builders generic to all relational databases? Or is there a need to create new ones for BigQuery?
• Once I have the data from a data source, how are they visualized on the dashboard?
• Is there a complete workbook of how things work behind the scene for reference?

[sambhxv]

follow-up, I have made some tweaks around the system and was able to list tables and fetch data from those tables using the underlying methods, had to add google-bigquery-dialect in order to make it work.

Here's the implementation for the same: Internal Insights

[nextchamp-saqib]

The new implementation looks good!

[sambhxv]

@nextchamp-saqib looks merge-able? should I raise a PR to insights/develop?