An easy-to-deploy Looker Studio Dashboard with alerting capabilities, showing usage and quota limits in an organization or folder.
Google Cloud enforces quotas on resource usage for project owners, setting a limit on how much of a particular Google Cloud resource your project can use. Each quota limit represents a specific countable resource, such as the number of API requests made per day to the number of load balancers used concurrently by your application.
Quotas are enforced for a variety of reasons:
We are introducing a new custom quota monitoring and alerting solution for Google Cloud customers.
Quota Monitoring Solution is a stand-alone application of an easy-to-deploy Looker Studio dashboard with alerting capabilities showing all usage and quota limits in an organization or folder.
*The data refresh rate depends on the configured frequency to run the application.
The architecture is built using Google Cloud managed services - Cloud Functions, Pub/Sub, Dataflow and BigQuery.
*Note 1 project will only have 1 app-code, but app-code can have more than 1 project.
e.g. If you have two rows in the csv file:
edge-retail-374401|pub-sub-example-394521, appcode1
edge-retail-374401, appcode2
edge-retail-374401 will end up with appcode2.
Terraform version >= 0.14.6 installed. Instructions to install terraform here
terraform -version
The output should look like:
Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/google v3.57.0
Note - Minimum required version v0.14.6. Lower terraform versions may not work.
In local workstation create a new directory to run terraform and store credential file
mkdir <directory name like quota-monitoring-dashboard>
cd <directory name>
Set default project in config to host project A
gcloud config set project <HOST_PROJECT_ID>
The output should look like:
Updated property [core/project].
Ensure that the latest version of all installed components is installed on the local workstation.
gcloud components update
Cloud Scheduler depends on the App Engine application. Create an App Engine application in the host project. Replace the region. List of regions where App Engine is available can be found here.
gcloud app create --region=<region>
Note: Cloud Scheduler (below) needs to be in the same region as App Engine. Use the same region in terraform as mentioned here.
The output should look like:
You are creating an app for project [quota-monitoring-project-3].
WARNING: Creating an App Engine application for a project is irreversible and the region
cannot be changed. More information about regions is at
<https://cloud.google.com/appengine/docs/locations>.
Creating App Engine application in project [quota-monitoring-project-1] and region [us-east1]....done.
Success! The app is now created. Please use `gcloud app deploy` to deploy your first app.
In local workstation, setup environment variables. Replace the name of the Service Account in the commands below
export DEFAULT_PROJECT_ID=$(gcloud config get-value core/project 2> /dev/null)
export SERVICE_ACCOUNT_ID="sa-"$DEFAULT_PROJECT_ID
export DISPLAY_NAME="sa-"$DEFAULT_PROJECT_ID
Verify host project Id.
echo $DEFAULT_PROJECT_ID
Create Service Account
gcloud iam service-accounts create $SERVICE_ACCOUNT_ID --description="Service Account to scan quota usage" --display-name=$DISPLAY_NAME
The output should look like:
Created service account [sa-quota-monitoring-project-1].
The following roles need to be added to the Service Account in the host project i.e. Project A:
Run following commands to assign the roles:
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.dataEditor" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/bigquery.jobUser" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/cloudfunctions.admin" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/cloudscheduler.admin" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/pubsub.admin" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/iam.serviceAccountUser" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/storage.admin" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/serviceusage.serviceUsageAdmin" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/cloudasset.viewer" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.networkViewer" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.viewer" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.notificationChannelEditor" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.alertPolicyEditor" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/logging.configWriter" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/logging.logWriter" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.viewer" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.metricWriter" --condition=None
gcloud projects add-iam-policy-binding $DEFAULT_PROJECT_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/iam.securityAdmin" --condition=None
SKIP THIS STEP IF THE FOLDER IS NOT THE TARGET TO SCAN QUOTA
If you want to scan projects in the folder, add following roles to the Service Account created in the previous step at the target folder A:
Set target folder id
export TARGET_FOLDER_ID=<target folder id like 38659473572>
Run the following commands add to the roles to the service account
gcloud alpha resource-manager folders add-iam-policy-binding $TARGET_FOLDER_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/cloudasset.viewer"
gcloud alpha resource-manager folders add-iam-policy-binding $TARGET_FOLDER_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.networkViewer"
gcloud alpha resource-manager folders add-iam-policy-binding $TARGET_FOLDER_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.viewer"
gcloud alpha resource-manager folders add-iam-policy-binding $TARGET_FOLDER_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/resourcemanager.folderViewer"
gcloud alpha resource-manager folders add-iam-policy-binding $TARGET_FOLDER_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.viewer"
Note: If this fails, run the commands again
SKIP THIS STEP IF THE ORGANIZATION IS NOT THE TARGET
If you want to scan projects in the org, add following roles to the Service Account created in the previous step at the Org A:
Set target organization id
export TARGET_ORG_ID=<target org id ex. 38659473572>
Run the following commands to add to the roles to the service account
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/cloudasset.viewer" --condition=None
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.networkViewer" --condition=None
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/compute.viewer" --condition=None
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/resourcemanager.folderViewer" --condition=None
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/resourcemanager.organizationViewer" --condition=None
gcloud organizations add-iam-policy-binding $TARGET_ORG_ID --member="serviceAccount:$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com" --role="roles/monitoring.viewer" --condition=None
Clone the Quota Management Solution repo
git clone https://github.com/google/quota-monitoring-solution.git quota-monitorings-solution
Change directories into the Terraform example
cd ./quota-monitorings-solution/terraform/example
Impersonate your host project service account and set environment variable using temporary token to authenticate terraform. You will need to make sure your user has the Service Account Token Creator role to create short-lived credentials.
gcloud config set auth/impersonate_service_account \
$SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com
export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)
TIP: If you get an error saying unable to impersonate, you will need to unset the impersonation. Have the role added similar to below, then try again.
# unset impersonation
gcloud config unset auth/impersonate_service_account
# set your current authenticated user as var
PROJECT_USER=$(gcloud config get-value core/account)
# grant IAM role serviceAccountTokenCreator
gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT_ID@$DEFAULT_PROJECT_ID.iam.gserviceaccount.com \
--member user:$PROJECT_USER \
--role roles/iam.serviceAccountTokenCreator \
--condition=None
Verify that you have these 3 files in your local directory:
Open terraform.tfvars file in your favourite editor and change values for the variables.
vi terraform.tfvars
For region
, use the same region as used for App Engine in earlier steps.
The variables source_code_base_url
, qms_version
, source_code_zip
and source_code_notification_zip
on the QMS module are used to download
the source for the QMS Cloud Functions from the latest GitHub release.
To deploy the latest unreleased code from a local clone of the QMS
repository, set qms_version
to main
Run terraform commands
terraform init
terraform plan
terraform apply
yes
This will:
Note: In case terraform fails, run terraform plan and terraform apply again
Stop impersonating service account (when finished with terraform)
gcloud config unset auth/impersonate_service_account
Initiate first job run in Cloud Scheduler.
Console
Click 'Run Now' on Cloud Job scheduler.
Note: The status of the ‘Run Now’ button changes to ‘Running’ for a fraction of seconds.
Terminal
gcloud scheduler jobs run quota-monitoring-cron-job --location <region>
gcloud scheduler jobs run quota-monitoring-app-alert-config --location <region>
To verify that the program ran successfully, check the BigQuery Table. The time to load data in BigQuery might take a few minutes. The execution time depends on the number of projects to scan. A sample BigQuery table will look like this:
Go to the Looker Studio dashboard template. A Looker Studio dashboard will look like this:
Make a copy of the template from the copy icon at the top bar (top - right corner)
Click on ‘Copy Report’ button without changing datasource options
This will create a copy of the report and open in Edit mode. If not click on ‘Edit’ button on top right corner in copied template:
Select any one table like below ‘Disks Total GB - Quotas’ is selected. On the right panel in ‘Data’ tab, click on icon ‘edit data source’ It will open the data source details ![ds_datasource_config_step_1]img/ds_datasource_config_step_1.png
Replace the BigQuery Project Id of your bq table, Dataset Id and Table Name to match your deployment. If you assigned app codes add a list of project ids in where clause from the csv file upload. Verify the query by running in BigQuery Editor for accuracy & syntax:
#For org level dashboard use the following query
SELECT
project_id,
added_at,
region,
quota_metric,
CASE
WHEN CAST(quota_limit AS STRING) ='9223372036854775807' THEN 'unlimited'
ELSE
CAST(quota_limit AS STRING)
END AS str_quota_limit,
SUM(current_usage) AS current_usage,
ROUND((SAFE_DIVIDE(CAST(SUM(current_usage) AS BIGNUMERIC), CAST(quota_limit AS BIGNUMERIC))*100),2) AS current_consumption,
SUM(max_usage) AS max_usage,
ROUND((SAFE_DIVIDE(CAST(SUM(max_usage) AS BIGNUMERIC), CAST(quota_limit AS BIGNUMERIC))*100),2) AS max_consumption
FROM
(
SELECT
*,
RANK() OVER (PARTITION BY project_id, region, quota_metric ORDER BY added_at DESC) AS latest_row
FROM
`[YOUR_PROJECT_ID].quota_monitoring_dataset.quota_monitoring_table`
) t
WHERE
latest_row=1
AND current_usage IS NOT NULL
AND quota_limit IS NOT NULL
AND current_usage != 0
AND quota_limit != 0
GROUP BY
project_id,
region,
quota_metric,
added_at,
quota_limit
# For app level dashboard use the following query replace PROJECT_ID with project_ids from csv file upload
SELECT
project_id,
added_at,
region,
quota_metric,
CASE
WHEN CAST(quota_limit AS STRING) ='9223372036854775807' THEN 'unlimited'
ELSE
CAST(quota_limit AS STRING)
END AS str_quota_limit,
SUM(current_usage) AS current_usage,
ROUND((SAFE_DIVIDE(CAST(SUM(current_usage) AS BIGNUMERIC), CAST(quota_limit AS BIGNUMERIC))*100),2) AS current_consumption,
SUM(max_usage) AS max_usage,
ROUND((SAFE_DIVIDE(CAST(SUM(max_usage) AS BIGNUMERIC), CAST(quota_limit AS BIGNUMERIC))*100),2) AS max_consumption
FROM
(
SELECT
*,
RANK() OVER (PARTITION BY project_id, region, quota_metric ORDER BY added_at DESC) AS latest_row
FROM
`[YOUR_PROJECT_ID].quota_monitoring_dataset.quota_monitoring_table`
) t
WHERE
latest_row=1
AND current_usage IS NOT NULL
AND quota_limit IS NOT NULL
AND current_usage != 0
AND quota_limit != 0
AND project-id IN ([PROJECT_ID1], [PROJECT_ID2]..)
GROUP BY
project_id,
region,
quota_metric,
added_at,
quota_limit
After making sure that query is returning results, replace it in the Data Studio, click on the ‘Reconnect’ button in the data source pane.
In the next window, click on the ‘Done’ button.
Once the data source is configured, click on the ‘View’ button on the top right corner. Note: make additional changes in the layout like which metrics to be displayed on Dashboard, color shades for consumption column, number of rows for each table etc in the ‘Edit’ mode.
Quota monitoring reports can be scheduled from the Looker Studio dashboard using ‘Schedule email delivery’. The screenshot of the Looker Studio dashboard will be delivered as a pdf report to the configured email Ids.
The alerts about services nearing their quota limits can be configured to be sent via email as well as following external services:
To configure notifications to be sent to a Slack channel, you must have the Monitoring Notification Channel Editor role on the host project.
You should now receive alerts in your Slack channel whenever a quota reaches the specified threshold limit.
The new version includes a fix that converts the data pull process to use the Montoring Query Language (MQL). This allows QMS to pull the limit and current usage at the exact same time, so reporting queries can be more tightly scoped, eliminating over reporting problems.
To upgrade existing installations:
Quota Monitoring Solution is a project based on open source contributions. We'd love for you to report issues, file feature requests, and send pull requests (see Contributing). Quota Monitoring Solution is not officially covered by the Google Cloud product support.