working with OSBA

[badalk]

Does keyvault flextool work with AKS OSBA? how do i create my secrets in key-vault and refer it in OSBA service manifests using keyvault flex tool? Any guidance?

[ritazh]

@badalk Sorry for the delay. Are you referring to using flexvol to mount certs used to deploy the OSBA service like here? https://github.com/Azure/open-service-broker-azure/blob/master/contrib/k8s/charts/open-service-broker-azure/templates/deployment.yaml#L131-L133

[badalk]

@ritazh the one you have shown is mounting the volume from kubernetes secret store. In Azure Kubernetes Service, secrets are not encrypted at rest and hence we want to completely avoid using it. We are able to do so with key-vault flex volume using option 2 (with pod identity approach and not service principal). Unfortunately, when we use OSBA to provision and connect other azure services, the secrets are automatically added to the kubernetes secret store. I was looking for a mechanism to add them to key-vault automatically instead and mount it as volume on the pod. Not sure if I am making sense. Let me know if there is a way OR you need any additional information

[ritazh]

@badalk This makes sense.

In Azure Kubernetes Service, secrets are not encrypted at rest

AKS does encrypt secrets-at-rest, but the keys are managed by AKS.

use OSBA to provision and connect other azure services, the secrets are automatically added to the kubernetes secret store.

I believe service catalog is responsible for creating/updating the k8s secrets created as a result of a service binding: https://github.com/kubernetes-incubator/service-catalog/blob/master/pkg/controller/controller_binding.go#L517

cc @jeremyrickard @carolynvs

[badalk]

@ritazh do you have any reference to documentation on AKS does encrypt secrets-at-rest, but the keys are managed by AKS. I am not able to find the reference to the doc where I read that secrets are just base64 encoded and not really encrypted at rest in etcd. I could be wrong and that is the reason I am trying to get this working. Would appreciate if you can share any reference to the doc

[ritazh]

AKS is a managed service that includes encryption at rest. I'm also not able to find any docs for this at the moment.

cc @seanmck @iainfoulds

[carolynvs]

I believe service catalog is responsible for creating/updating the k8s secrets created as a result of a service binding

Service Catalog is capable of either creating a new secret for you or injecting values into an existing secret.

[jemag]

I am sorry if this a dumb question, but if secrets are encrypted at rest with AKS, does it mean that using straight secrets with AKS would be secure enough to use in a production setup?

If so, what are the security advantages of using Azure Key Vault and this project flexvol?

[ritazh]

If you already have an enterprise-grade secrets store for managing and storing your application secrets, e.g. Azure Key vault, and you want Kubernetes to use the same source of truth for secrets, then this project provides a way to retrieve sensitive data from the store into your Kubernetes applications using a volume.

[jemag]

Thank you for this clarification.

We are interested in using some 3rd party helm charts and since they already include secrets, we were worried about not being able to use those if AKS was using only base 64 encoding (which, thanks to your comments, seems to not be the case). This might be a concern for other teams, I really wish this was more clearly specified in the documentation.

Azure Key Vault does look like a good solution for a single source of truth, but I am not sure how someone would proceed with 3rd party Helm charts secrets. Also, it seems like it would be simpler to pass environment variables for spring boot applications through secrets, rather than having to mount a volume and fetching the secrets there (although probably a bit less secure).

[ritazh]

Thanks for the feedback. Will work with docs team to provide better documentation. For ENV vars, take a look at the discussions in this issue and possible workarounds: https://github.com/Azure/kubernetes-keyvault-flexvol/issues/28

[badalk]

@iainfoulds @seanmck any guidance on the documentation which clarifies how secrets are stored in AKS?

[iainfoulds]

@badalk - @ritazh just mentioned five minutes beforehand about working with docs, so nothing more to share just yet :) We'll prioritize the request against existing doc needs that we have in our backlog. I don't have an ETA on that.

[badalk]

Thanks @ianfoulds and @ritazh . While documentation gets updated, Can I assume what Ritazh mentioned (AKS does encrypt secrets-at-rest, but the keys are managed by AKS) to make certain decisions.

[ritazh]

@badalk Yes. and we will let you know when we have updated our Microsoft docs. Thank you for your patience!

[KevinJCross]

Does written in unencrypted text to a temp fs really count as encrypted at rest?

[jemag]

Does written in unencrypted text to a temp fs really count as encrypted at rest?

Pretty sure they mean encrypted at rest on on the master node. The behavior you are describing is once you use a secret within a pod.