Open om3171991 opened 2 years ago
Hi,
I looked at the link you gave which specified these instructions:
curl -X POST 'https://login.microsoftonline.com/
/oauth2/v2.0/token' -d 'grant_type=client_credentials&client_id= &client_secret= &scope=https://vault.azure.net/.default'
followed by
curl -X GET 'https://
.vault.azure.net/secrets/ ?api-version=7.2' -H "Authorization: Bearer "
Can you verify that the name of your KeyVault instance is azure-vault
? By setting the avp.kubernetes.io/path
annotation to that value, you're telling AVP your KeyVault instance is located at the URL: azure-vault.vault.azure.net
: https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L32.
If that value is correct, maybe your instance is taking longer than 10 seconds to respond with the secrets? https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L34. You can try timing the response in your curl
test to see, or try increasing the timeout in the AVP code and then recompile and load into the repo-server and testing that way. Not sure how likely this is to be the problem.
Otherwise, I'll have to look deeper (and could use your eyes on the code here: https://github.com/argoproj-labs/argocd-vault-plugin/blob/main/pkg/backends/azurekeyvault.go#L1 since I don't use the Azure KeyVault product myself).
Hi @jkayani,
I double-checked on timeouts and with MSFT curls, we are getting responses in a max of 2 seconds and the name of the vault is correct. Don't have much idea of Go but will try to recompile code with a higher timeout and request if can also lookout.
Hey @om3171991 do you have more than 25 secrets in your vault?
I looked a bit at the code and this line, where it says nil
is an option that can be set to how many secrets that maximum should be fetched:
Docs for the final function GetSecrets
:
// GetSecrets the Get Secrets operation is applicable to the entire vault. However, only the base secret identifier and
// its attributes are provided in the response. Individual secret versions are not listed in the response. This
// operation requires the secrets/list permission.
// Parameters:
// vaultBaseURL - the vault name, for example https://myvault.vault.azure.net.
// maxresults - maximum number of results to return in a page. If not specified, the service will return up to
// 25 results.
I dont use Azure key vault anymore but setting that to a higher number might fix it. Please tell if you have more than 25 secrets.
@om3171991 are you still having issues?
I'm having this issue from an ArgoCD plugin sidecar. I'm migrating from argocd-cm plugins to sidecar plugins and all of my sidecars that require argocd-vault-plugin have this issue.
https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/troubleshoot-key-vault-csi-secrets-store-csi-driver#cause-2-keyvaultbaseclientgetsecret-failure-sending-request-statuscode0-error This issue is related to the repo server not able to access key vault. Hopefully the link will help. I would check for networking policies on your pods and see if something is preventing that.
Describe the bug While trying to fetch secret from Azure vault using argocd-vault-plugin, we are facing below error message
keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
cmd run to get secret on repo server pod: cat example.yaml |argocd-vault-plugin -c vault.yaml generate - Output : keyvault.BaseClient#GetSecret: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded example.yaml
kind: Secret apiVersion: v1 metadata: name: example-annotation annotations: avp.kubernetes.io/path: "azure-vault" type: Opaque data: username: <USER-SECRET>
cat vault.yaml
AZURE_CLIENT_SECRET: <Removed> AZURE_CLIENT_ID: <Removed> AZURE_TENANT_ID: <Removed> AVP_TYPE: azurekeyvault
Expected behavior The secret value should be fetched from the Azure vault
Additional context We have looked Azure portal for a similar issue and tried curl from repo-server as asked (https://docs.microsoft.com/en-us/azure/aks/csi-secrets-store-troubleshooting#keyvaultbaseclientgetsecret-failure-sending-request-statuscode0--original-error-context-canceled), I see values are coming from the vault as expected.
Please help in debugging.