[Question] - How to enable audit devices to stdout and save in Stackdriver Logs?

[samuelbaruffi]

Hello, First of all, thanks for the awesome tutorial. It is very handy.

We have implemented this in our production cluster and were having issues getting the audit device logs to Stackdriver logs.

I have enabled the audit device to stdout by doing the following:

vault audit enable file file_path=stdout

Which I can confirm that is outputting to stdout on the vault container, if I check the logs with:

kubectl logs vault-0 -f vault

But unfortunately those logs are not being saved in Stackdriver for some reason, and I was not able to find more info on how to enable or troubleshoot it. See picture below for my stackdriver log on the vault container:

Thanks in advance for the help.

Sam.

[sethvargo]

Hi @samuelbaruffi

Can you share more of that screenshot? The logs should be there, including the Vault startup logs. Can you make sure you're looking at the correct container? What does kubectl get logs show for that container?

[samuelbaruffi]

Thanks for the quick reply @sethvargo .

See the full screenshot below (hiding few fields for security reasons):

The logs for the vault-init containers are being saved to Stackdriver, but for the vault container it does not seem they are. See screenshot below fir the vault-init container in the pod that is working on Stackdriver:

If I run kubectl logs vault-0 -f vault I'm able to see all the audit logs.

Thanks for the help!

[sethvargo]

Hmm - that's really weird, since they are deployed the same. Are you able to reproduce it on a new cluster?

[samuelbaruffi]

I'd have to try creating a new cluster and building Vault again.

I'll post the results once I am able to replicate the environment in a new cluster.

Let me know if you find anything meanwhile.

Thank you.

[sethvargo]

I'm not able to reproduce it on my end. If you're familiar with Terraform, github.com/sethvargo/vault-on-gke is a one-command version of this same thing.

[samuelbaruffi]

Thank you @sethvargo,

I'll try to use the Terrraform script for my testing.

For now I'll go ahead and close this ticket.

Sam.

[MaxDiOrio]

I'm seeing issues with logging with the Terraform script. Absolutely 0 Kubernetes logs in Stackdriver. But viewing kubectl logs for the Vault container shows the audit logs properly.

For a cluster created "manually" through the GCloud UI, you can see the K8S logs:

For the Vault cluster created through the Terraform, nothing.