Weights & Biases Google Module

This is a Terraform module for provisioning a Weights & Biases Cluster on Google Cloud. Weights & Biases Local is our self-hosted distribution of wandb.ai. It offers enterprises a private instance of the Weights & Biases application, with no resource limits and with additional enterprise-grade architectural features like audit logging and single sign-on.

About This Module

Pre-requisites

This module is intended to run in an Google Cloud account with minimal preparation, however it does have the following pre-requisites:

Terrafom version >= 1

Credentials / Permissions

Google Services Used

Google SQL Cloud (MySQL)
Google Kubernetes Engine
Google Storage Bucket
Google PubSub
Google Managed Certificates
Google Cloud DNS

How to Use This Module

Ensure account meets module pre-requisites from above.
Create a Terraform configuration that pulls in this module and specifies values of the required variables:

provider "google" {
  project = "<desired google project>"
  region = "<desired google region>"
  zone = "<desired google zone>"
}

module "wandb" {
  source    = "<filepath to cloned module directory>"
  namespace = "<prefix for naming google resources>"
}

Run terraform init and terraform apply

Examples

We have included documentation and reference examples for common installation scenarios, as well as examples for supporting resources that lack official modules.

Public Instance with HTTPS using Cloud DNS

Requirements

Name	Version
terraform	~> 1.0
google	~> 5.30
helm	~> 2.10
kubernetes	~> 2.23
time	0.11.2

Providers

Name	Version
google	~> 5.30

Modules

Name	Source	Version
app_gke	./modules/app_gke	n/a
app_lb	./modules/app_lb	n/a
database	./modules/database	n/a
gke_app	wandb/wandb/kubernetes	1.14.1
kms	./modules/kms	n/a
networking	./modules/networking	n/a
private_link	./modules/private_link	n/a
project_factory_project_services	terraform-google-modules/project-factory/google//modules/project_services	~> 14.0
redis	./modules/redis	n/a
service_accounts	./modules/service_accounts	n/a
sleep	matti/resource/shell	1.5.0
storage	./modules/storage	n/a
wandb	wandb/wandb/helm	1.2.0

Resources

Name	Type
google_client_config.current	data source
google_compute_forwarding_rules.all	data source

Inputs

Name	Description	Type	Default	Required
allowed_inbound_cidrs	Which IPv4 addresses/ranges to allow access. This must be explicitly provided, and by default is set to ["*"]	`list(string)`	[ "*" ]	no
allowed_project_names	A map of allowed projects where each key is a project number and the value is the connection limit.	`map(number)`	`{}`	no
app_wandb_env	Extra environment variables for W&B	`map(string)`	`{}`	no
bucket_name	Use an existing bucket.	`string`	`""`	no
create_private_link	Whether to create a private link service.	`bool`	`false`	no
create_redis	Boolean indicating whether to provision an redis instance (true) or not (false).	`bool`	`false`	no
create_workload_identity	Flag to indicate whether to create a workload identity for the service account.	`bool`	`false`	no
database_machine_type	Specifies the machine type to be allocated for the database	`string`	`"db-n1-standard-2"`	no
database_sort_buffer_size	Specifies the sort_buffer_size value to set for the database	`number`	`67108864`	no
database_version	Version for MySQL	`string`	`"MYSQL_8_0_31"`	no
deletion_protection	If the instance should have deletion protection enabled. The database / Bucket can't be deleted when this value is set to `true`.	`bool`	`true`	no
disable_code_saving	Boolean indicating if code saving is disabled	`bool`	`false`	no
domain_name	Domain for accessing the Weights & Biases UI.	`string`	`null`	no
enable_stackdriver	n/a	`bool`	`false`	no
force_ssl	Enforce SSL through the usage of the Cloud SQL Proxy (cloudsql://) in the DB connection string	`bool`	`false`	no
gke_machine_type	Specifies the machine type to be allocated for the database	`string`	`"n1-standard-4"`	no
gke_node_count	n/a	`number`	`2`	no
ilb_proxynetwork_cidr	Internal load balancer proxy subnetwork	`string`	`"10.127.0.0/24"`	no
labels	Labels to apply to resources	`map(string)`	`{}`	no
license	Your wandb/local license	`string`	n/a	yes
local_restore	Restores W&B to a stable state if needed	`bool`	`false`	no
namespace	String used for prefix resources.	`string`	n/a	yes
network	Pre-existing network self link	`string`	`null`	no
oidc_auth_method	OIDC auth method	`string`	`"implicit"`	no
oidc_client_id	The Client ID of application in your identity provider	`string`	`""`	no
oidc_issuer	A url to your Open ID Connect identity provider, i.e. https://cognito-idp.us-east-1.amazonaws.com/us-east-1_uiIFNdacd	`string`	`""`	no
oidc_secret	The Client secret of application in your identity provider	`string`	`""`	no
other_wandb_env	Extra environment variables for W&B	`map(string)`	`{}`	no
parquet_wandb_env	Extra environment variables for W&B	`map(string)`	`{}`	no
psc_subnetwork_cidr	Private link service reserved subnetwork	`string`	`"192.168.0.0/24"`	no
public_access	Whether to create a public endpoint for wandb access.	`bool`	`true`	no
redis_reserved_ip_range	Reserved IP range for REDIS peering connection	`string`	`"10.30.0.0/16"`	no
redis_tier	Specifies the tier for this Redis instance	`string`	`"STANDARD_HA"`	no
resource_limits	Specifies the resource limits for the wandb deployment	`map(string)`	{ "cpu": null, "memory": null }	no
resource_requests	Specifies the resource requests for the wandb deployment	`map(string)`	{ "cpu": "2000m", "memory": "2G" }	no
size	Deployment size for the instance	`string`	`null`	no
ssl	Enable SSL certificate	`bool`	`true`	no
stackdriver_sa_name	n/a	`string`	`"wandb-stackdriver"`	no
subdomain	Subdomain for accessing the Weights & Biases UI. Default creates record at Route53 Route.	`string`	`null`	no
subnetwork	Pre-existing subnetwork self link	`string`	`null`	no
use_internal_queue	Uses an internal redis queue instead of using google pubsub.	`bool`	`false`	no
wandb_image	Docker repository of to pull the wandb image from.	`string`	`"wandb/local"`	no
wandb_version	The version of Weights & Biases local to deploy.	`string`	`"latest"`	no
weave_wandb_env	Extra environment variables for W&B	`map(string)`	`{}`	no

Outputs

Name	Description
address	n/a
bucket_name	Name of google bucket.
bucket_queue_name	Pubsub queue created for google bucket file upload events.
cluster_ca_certificate	Certificate of the kubernetes (GKE) cluster.
cluster_client_certificate	n/a
cluster_client_key	n/a
cluster_endpoint	Endpoint of the kubernetes (GKE) cluster.
cluster_id	ID of the kubernetes (GKE) cluster.
cluster_name	n/a
cluster_node_pool	Default node pool where Weights & Biases should be deployed into.
cluster_self_link	Self link of the kubernetes (GKE) cluster.
database_connection_string	Full database connection string. You must be in the VPC to access the database.
database_instance_type	n/a
fqdn	The FQDN to the W&B application
gke_node_count	n/a
gke_node_instance_type	n/a
private_attachement_id	n/a
sa_account_email	This output provides the email address of the service account created for workload identity, if workload identity is enabled. Otherwise, it returns null
service_account	Weights & Biases service account used to manage resources.
standardized_size	n/a
url	The URL to the W&B application

Migrations

3.x -> 4.x

3.6.0 introduced a change in the Google Provider that isn't backwards compatible with prior versions. Nothing needs to be done to upgrade, but it is not backwards compatible.

wandb / terraform-google-wandb

readme