Feature: Add support for multi-project setup

[dprokop]

In https://github.com/grafana/google-bigquery-datasource/issues/101 chrisleavoy described an architecture that surfaced several issues current user experience suffers from.

The limitations are caused by the false assumption that when using GCE authentication both the VM running Grafana and BigQuery data are "located" under the same GCP project. A good example of an exception to that rule is well described in https://github.com/grafana/google-bigquery-datasource/issues/101#issuecomment-1101420096 and illustrated in the following diagram:

The following PRs:

• 102

• 103

introduced an option to configure a GCP project that will be responsible for running BigQuery jobs.

While queries can be performed cross-project thanks to the mentioned changes, there is no UI for the user to choose datasets from different projects when using either Visual Query Builder or Code Editor. It's especially problematic with the VQB, as data source will always ask for GCE's default project's datasest, which may not contain any BigQuery datasets.

In order to support this scenario we need to introduce several changes to both data source implementation and the user interface:

• [x] API: Add a method for retrieving projects from GCP:
  • [x] Integrate with Resource Manager API to allow listing projects within GCP. SDK: https://pkg.go.dev/google.golang.org/api/cloudresourcemanager.
  • [x] Allow multiple resource routes when creating HTTP client. We currently have fixed reference to BigQuery resource route: https://github.com/grafana/google-bigquery-datasource/blob/main/pkg/bigquery/http_client.go#L30. Take a look at Google Cloud Monitoring data source implementation for a reference: https://github.com/grafana/grafana/blob/main/pkg/tsdb/cloudmonitoring/httpclient.go
  • [x] Similarily to BigQuery API client create a Resource Manager API client (ref. https://github.com/grafana/google-bigquery-datasource/blob/main/pkg/bigquery/datasource.go#L111)
  • [x] Update documentation with an information that Resource Manager API needs to be enabled in GCP project (ref. https://grafana.com/docs/grafana/latest/datasources/google-cloud-monitoring/#google-cloud-monitoring-specific-data-source-configuration)
• [x] VQB: Add the possibility to choose a project when creating a query. The selected project should be used for listing datasets.
• [x] CodeEditor: Add autocomplete for multiple projects (and datasets within those). Currently datasets, tables and coluns API assume the provided project is a "default project" - we need to allow providing an information to those methods (on the client side) what is the project context.
• [x] Validate whether or not the setup above is also possible with JWT Token authentication, when Grafana instance is running outside of GCP.

[dprokop]

@zoltanbedi @oscarkilhed: cross-project service account/IAM details: https://stackoverflow.com/questions/35479025/cross-project-management-using-service-account

[dprokop]

@chrisleavoy this feature is now released with V1.1.0, hope it's gonna make your setup work even better now!