Operator | Optionally Prevent Logging of Sensitive info in Plain Text in wandb-controller-pod Logs

[vijay-wandb]

Sensitive data, like passwords, should either be omitted or masked (e.g., replaced with **) in the wandb-controller-pod logs. Optionally, provide a config setting that allows administrators to enable or disable the logging of sensitive information.

[abhinavg6]

@velotioaastha @amanpruthi - Looks like @zacharyblasczyk has addressed this in the latest version of helm. The team will retest that once it's available.

[abhinavg6]

@vijay-wandb - Were you able to retest this?

[vijay-wandb]

@abhinavg6

Only "accessKey":"***" is masked, fields like secretKey, license, database password and redis password are still visible. Please see example below. I manually partially masked those fields with xxx

{"level":"info","ts":"2024-09-16T12:49:51Z","msg":"Active spec found","controller":"weightsandbiases","controllerGroup":"apps.wandb.com","controllerKind":"WeightsAndBiases","WeightsAndBiases":{"name":"wandb","namespace":"wandb-aws"},"namespace":"wandb-aws","name":"wandb","reconcileID":"4d3e24fb-a71c-490e-a90f-ca1aa5cec7f5","spec":{"metadata":{"channelId":"b56e1972-3c78-4de0-af90-e3597bb0785a","channelInheritsFrom":"","channelName":"Stable","releaseCreatedAt":"2024-09-12T19:00:06.196Z","releaseExtends":"","releaseId":"0be1b3a7-d85a-49e3-9eb4-140f0c0e8d2f","releaseName":"v202409-12.19"},"chart":{"url":"https://charts.wandb.ai","name":"operator-wandb","version":"0.17.5","password":"","username":""},"values":{"app":{"image":{"repository":"wandb/local","tag":"0.59.1"}},"console":{"image":{"repository":"wandb/console","tag":"2.12.2"}},"global":{"bucket":{"accessKey":"***","name":"vijay-wandb","provider":"s3","region":"us-east-1","secretKey":"djswjHk5iFFJIwQQS8mBGBdgIhfYyXusXoTydZxx"},"extraEnv":{"AWS_REGION":"us-east-1"},"host":"https://wandb-aws.world-iq.com","license":"eyJhbGciOiJSUzI1NiIsImtpZCI6InUzaHgyQjQyQWhEUXM1M0xQY09yNnZhaTdoSlduYnF1bTRZTlZWd1VwSWM9In0.eyJjb25jdXJyZW50QWdlbnRzIjoxMCwiZGVwbG95bWVudElkIjoiOTFkNjgyMWEtNTJjZC00OGQxLThmNjYtMjlmODI4NWYwNzU4IiwibWF4VXNlcnMiOjEwMCwibWF4Vmlld09ubHlVc2VycyI6MCwibWF4U3RvcmFnZUdiIjoxMDAwMDAwLCJtYXhUZWFtcyI6NTAsIm1heFJlZ2lzdGVyZWRNb2RlbHMiOjUsImV4cGlyZXNBdCI6IjIwMjgtMTItMjVUMDQ6NTk6NTkuOTk5WiIsImZsYWdzIjpbIlNDQUxBQkxFIiwibXlzcWwiLCJzMyIsInJlZGlzIiwiTk9USUZJQ0FUSU9OUyIsInNsYWNrIiwibm90aWZpY2F0aW9ucyIsIk1BTkFHRU1FTlQiLCJvcmdfZGFzaCIsImF1dGgwIiwiY29sbGVjdF9hdWRpdF9sb2dzIiwicmJhYyJdLCJ0cmlhbCI6dHJ1ZSwiYWNjZXNzS2V5IjoiMTIxMzg0MTctYmM4OC00YmNjLWJiYzUtMGM2NTY5NDMzMzE2Iiwic2VhdHMiOjEwMCwidmlld09ubHlTZWF0cyI6MCwidGVhbXMiOjUwLCJyZWdpc3RlcmVkTW9kZWxzIjo1LCJzdG9yYWdlR2lncyI6MTAwMDAwMCwiZXhwIjoxODYxMzMzMTk5fQ.G6fhzyGbVHtOuAFbjGArZpp-9sSrzOVezkGau-siElFpFGmDEQYdwsEPnuPs7vkxxx","mysql":{"database":"wandb_aws","host":"vijay-jwt-testing-rds.cluster-cbr7jjaqsamz.us-east-1.rds.amazonaws.com","password":"xxdiyaxx","port":3306,"user":"admin"},"operator":{"apiVersion":"apps.wandb.com/v1","namespace":"wandb-aws"},"redis":{"caCert":"","host":"redis-15242.c241.us-east-1-4.ec2.cloud.redislabs.com","parameters":{},"password":"pOvNTpE7WE9bPShCoLNSviNPQULmxx","port":15242}},"mysql":{"install":false},"parquet":{"image":{"repository":"wandb/local","tag":"0.59.1"}},"redis":{"install":false},"weave":{"image":{"repository":"wandb/local","tag":"0.59.1"}},"weave-trace":{"image":{"repository":"wandb/weave-trace","tag":"0.0.4"}}}}}

[abhinavg6]

@zacharyblasczyk - Thoughts on Vijay's test?

cc @danielpanzella

[zacharyblasczyk]

https://github.com/wandb/operator/blob/main/pkg/wandb/spec/spec.go#L139

We need to update the list here.

[vijay-wandb]

@zacharyblasczyk @velotioaastha

Here is the desired list of sensitiveKeys

secret accessKey license Database -> password Redis -> password

[amanpruthi]

https://github.com/wandb/operator/pull/31