This is not an officially supported Google product.
Spindle is a QA and feature adoption dashboard for Display & Video 360 (DV360) advertisers. It can be used to quickly identify financial (e.g. overspend) or reputational (e.g. brand safety) risk, and validate best practice across multiple advertisers. Spindle uses a set of predefined ‘flags’ to stack rank DV360 entities according to potential risk, but is fully configurable, and can be customised to check for specific best practice or flags (e.g. adherence to naming conventions, brand safety controls, audience, geo targeting, etc).
Spindle utilises Orchestra, an open source data orchestration tool developed on Google Cloud Composer. This repository is intended to streamline the deployment of Spindle and the necessary components from Orchestra, to a new or existing Cloud Composer environment.
The easiest and quickest way to deploy Spindle, is by using this deployment script. New deployments should take approximately 25 minutes to complete. Deployments to existing Composer environments should finish in under 5 mins.
This script will use the default service credentials for Compute Engine ("Compute Engine default service account"). These service accounts have access to all Cloud APIs enabled for your Cloud project, making them a good fit for Spindle/Orchestra.
:exclamation: If deploying to an existing Composer environment, ensure it has DoubleClick Bid Manager included in the 'Google API scopes' or Airflow connection.
Your Compute Engine service account email (similar to XX-compute@developer.gserviceaccount.com) will need to be setup as a DV360 user so that it can access data from your DV360 account. DV360 partner-level access is required to create a new user. New DV360 users should be created using this configuration:
git clone https://github.com/google/spindle-dv360
spindle.config
For deployments to a new Composer instance, spindle.config
should follow this template:
Field | Description | Example | |
---|---|---|---|
cloud_project_id | Required | Google Cloud Project ID | my-gcp-project1 |
bq_dataset | Required | BigQuery destination dataset name | spindle_clientname |
composer_gcs_bucket | Optional | Cloud Storage bucket URI. Do not include leading 'gs://' or trailing '/dags' | Leave blank for new Composer instances |
install_composer | Required | Indicates whether to create a new Composer instance | true |
composer_location | Required | Compute Engine region | europe-west1 |
composer_zone | Required | Compute Engine zone | europe-west1-c |
Your completed spindle.config
should look similar to:
{
"cloud_project_id": "my-gcp-project1",
"bq_dataset": "spindle_clientname",
"composer_gcs_bucket": "",
"install_composer": "true",
"environment_name": "clientname-spindle",
"composer_location": "europe-west1",
"composer_zone": "europe-west1-c"
}
For deployments to an existing Composer instance:
Field | Description | Example | |
---|---|---|---|
cloud_project_id | Required | Google Cloud Project ID | my-gcp-project1 |
bq_dataset | Required | BigQuery destination dataset name | spindle_clientname |
composer_gcs_bucket | Required | Cloud Storage bucket URI. Do not include leading 'gs://' or trailing '/dags' | europe-west2-spindle-42cn4da-bucket |
install_composer | Required | Indicates whether to create a new Composer instance | false |
Your completed spindle.config
should look similar to:
{
"cloud_project_id": "my-gcp-project1",
"bq_dataset": "spindle_clientname",
"composer_gcs_bucket": "europe-west2-spindle-42cn4da-bucket",
"install_composer": "false",
"environment_name": "",
"composer_location": "",
"composer_zone": ""
}
spindle.config
, execute the following in the Cloud Shell:cd spindle-dv360
. deploy.sh spindle.config
If you've used this script to create a new Composer instance, most of the Airflow configuration has been done for you, but you'll need to update your DV360 Partner IDs...
If you're deploying to an existing Composer environment, Airflow variables must be created manually, detailed here, to prevent overwriting or breaking an existing configuration.
By default, the Spindle DAG is configured to run at 10PM daily. This means that your new BigQuery tables will not be generated until the first run has finished. Once your pipeline has finished for the first time, and BigQuery dataset populated, you should proceed with this setup.
:warning: Warning: manually triggering the DAG outside the schedule (10pm) may result in duplicate rows in your dataset.
:warning: Warning: sharing the Data Studio dashboard with other users will allow them to view all data in the dashboard, regardless of whether they have the underlying DV360 account access permissions to view the DV360 Partner(s) surfaced in the dashboard. Access to the dashboard should be carefully managed and controlled.
Orchestra is a framework that allows powerful API access to your data. Liability for how you use that data is your own. It is important that all data you keep is secure and that you have legal permission to work and transfer all data you use. Orchestra can operate across multiple Partners, please be sure that this access is covered by legal agreements with your clients before implementing Orchestra. This project is covered by the Apache License.
Apache 2.0