search

bank-vaults / bank-vaults

A Vault swiss-army knife: A CLI tool to init, unseal and configure Vault (auth methods, secret engines).
https://bank-vaults.dev
Apache License 2.0
2.02k stars 465 forks source link

bank-vaults support for existing HashiCorp Vault users #863

Open Mel-Github opened 4 years ago

Mel-Github commented 4 years ago

Is your feature request related to a problem? Please describe. Some of the existing users have already an existing HashiCorp Vault instance running in their environment. However, currently to leverage the bank-vaults solutions we need to create a new vault instance via the custom crd.

Describe the solution you'd like I would like to still leverage the vault bank solution but running on top of my existing Hashicorp vault instance as there are hundreds of secrets already in my existing vault system.

mazzy89 commented 4 years ago

I've explored this solution right today and here my findings that can be obvious but useful to report:

These two commands make the Vault operations easy and straightforward.

Am I missing anything here? I would be keen to make the adjustments where necessary so this nice and handy tool can be used also in a non-kubernetes environment or in existing Vault clusters non-running on Kubernetes.

mazzy89 commented 4 years ago

Would make it make sense here to change the line https://github.com/banzaicloud/bank-vaults/blob/a1c1d5f4e76bd73eb96188eabdeedb6905b6d8bf/cmd/bank-vaults/unseal.go#L106

in

if !v.isInit() && !unsealConfig.raftSecondary {
...

in this way instead, to check the POD_NAME we would check the state using the following API request https://www.vaultproject.io/api/system/init.html

@bonifaido Do you see anything wrong in this?

bonifaido commented 4 years ago

Hi @mazzy89,

yes, this is right, you got the point.

For the second question, I'm not sure if just calling the API alone would be enough here, how would you decide in case of the first instance that it should be initialized or joining the cluster?

mazzy89 commented 4 years ago

I was thinking that since there is a built-in back-off mechanism, the init would work in a way that the first instance that is up, initializes the cluster and all the other instances just follow joining the Raft cluster.

bonifaido commented 4 years ago

And how do you decide in the case of the second instance that it should join instead of initing?

mazzy89 commented 4 years ago

Each instance would check if the cluster is already initialized. If so then join otherwise init. But maybe @bonifaido I'm missing something here in the big picture. If so then forgive me.

bonifaido commented 4 years ago

No, I'm okay with such an implementation if it makes the usage more agnostic! I still have a Raft test that I would like to add the acceptance test part (https://github.com/banzaicloud/bank-vaults/pull/829) if that passes I welcome any kind of improvements ;)

mazzy89 commented 4 years ago

@bonifaido cool. then I can help here because I can test bank-vaults on Vault cluster with Raft and have quick feedbacks.

mazzy89 commented 4 years ago

@bonifaido do you see any valid points in storying the leader IP close to KMS keys stored in the cloud? in this way, we could automate a bit the raft join and avoid to pass as input the leader address.

bonifaido commented 4 years ago

I have no problems with that, maybe we could use the keys to check if they exist for init/join as well.

mazzy89 commented 4 years ago

the only problem is when the mode file is used. in that case checking the existence of that key is harder.

bonifaido commented 4 years ago

The file backend is only for testing purposes.

mazzy89 commented 4 years ago

For the next PR about automating the retrieval of the Raft Leader, would it make sense here to get the leader addrress by querying the endpoint https://www.vaultproject.io/api-docs/system/leader?

so still leave the flag raft-leader-address and in case it is passed use that as leader. in case it is not passed query the endpoint /sys/leader

mazzy89 commented 4 years ago

@bonifaido thank you to help here to have pushed the changes and have it in the master in just a few days. that was great.

Any feedbacks on this https://github.com/banzaicloud/bank-vaults/issues/863#issuecomment-584186920?

bonifaido commented 4 years ago

I'm absolutely okay with that, where would you store the initial API address to ask /sys/leader? In the unseal KV backend, right?

bonifaido commented 4 years ago

However, I'm not sure if this is the right thread (issue) for these developments :)

mazzy89 commented 4 years ago

I'm absolutely okay with that, where would you store the initial API address to ask /sys/leader? In the unseal KV backend, right?

Yeah storing there would be fine. @bonifaido going to open another thread. I have some further questions I'd like to discuss here.

bonifaido commented 4 years ago

@Mel-Github would you like to use the webhook? If yes, you don't need a new Vault instance, you can use it simply via the webhook, you just need to configure the K8S auth backend in that instance for the cluster where the webhook is running.

russellseymour commented 4 years ago

@bonifaido Is there an example of getting the webhook to communicate with a separate Vault instance using the K8s auth backend?

I am in the position where I already have Vault up and running and want to use the webhook in my pods to get secrets from the vault, however I am having problems getting it working :(.

I thought that I would be able to use the External vault option in annotations. However after I found this thread I am not so sure now.

The other thing is that I am using helm to deploy the webhook, however it attempts to start up its own vault.

So in summary my questions are:

Thanks lots

matyix commented 4 years ago

Hello @russellseymour

Yes, it works as long as Vault is accessible, you have the K8s SA auth enabled in Vault and you have the right annotations. You can point to any Vault inside/outside K8s using similar annotations like here: https://banzaicloud.com/docs/bank-vaults/mutating-webhook/#running-webhook-and-vault-in-different-k8s-cluster, #6.

You might want to check similar questions on our community Slack as well (this has been asked/discussed many times like here): https://community-banzaicloud.slack.com/archives/CFJJW9L94/p1580893876140400

russellseymour commented 4 years ago

Thank you very much @matyix

I will have a look at the the inks you posted. I think I am almost there :-)

Mel-Github commented 4 years ago

@Mel-Github would you like to use the webhook? If yes, you don't need a new Vault instance, you can use it simply via the webhook, you just need to configure the K8S auth backend in that instance for the cluster where the webhook is running.

@bonifaido sorry for the late reply. i would like to give it a try. However i am would definitely needs some guidance to setup that integration.

matyix commented 4 years ago

@Mel-Github the integration is super simple - try this above (https://github.com/banzaicloud/bank-vaults/issues/863#issuecomment-591508969) and if you have issues let us know (here or Slack).

@russellseymour did you succeed with this?

Alternatively you can try the Pipeline platform and check out how and automated integration works (it uses Bank-Vaults under the hood). It's an integrated service, and it comes out of the box and available for any cluster created by Pipeline or imported into Pipeline.

caiobegotti commented 3 years ago

Hey folks, I just did read the whole issue but it's still unclear to me whether it's possible to use an existing (too-complicated-to-fully-migrate-now) Vault setup with Bank-Vaults or not. If it is, is there any detailed documentation on this with examples maybe? It's really not clear if it's doable based on the links shared here so far. I'm particularly interested in the backup/restore support with Velero that Bank-Vaults has. Thanks for all the hard work nonetheless!