Secret from Vault is double base64 encoded

[devth]

Added log at https://github.com/itscontained/secret-manager/blob/999e7d0a89f8fbcb5d2e3df11a3b75632f8024e0/pkg/internal/vault/vault.go#L150

    fmt.Printf("readSecret secret data: %+v\n", secretData)

Verified secret is not base64 encoded via logs:

2020-10-12T22:15:30.833546792Z readSecret secret data: map[data:map[another_one:awesome yessssssssss:this is secret yetibot_secret:secret value] metadata:map[created_time:2020-10-09T19:08:57.676679845Z deletion_time: destroyed:false version:3]]

readSecret secret data: map[data:map[embedded-enabled:false observers-enabled:false] metadata:map[created_time:2020-09-21T15:41:58.434072039Z deletion_time: destroyed:false version:1]]

Then looked at the K8S Secret that secret-manager created:

± k get secret yetibot-es -ojson | jq -r '.data'
{
  "another_one": "WVhkbGMyOXRaUT09",
  "embedded-enabled": "Wm1Gc2MyVT0=",
  "observers-enabled": "Wm1Gc2MyVT0=",
  "yessssssssss": "ZEdocGN5QnBjeUJ6WldOeVpYUT0=",
  "yetibot_secret": "YzJWamNtVjBJSFpoYkhWbA=="
}

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]'
WVhkbGMyOXRaUT09

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]' | base64 -d
YXdlc29tZQ==%

± k get secret yetibot-es -ojson | jq -r '.data["another_one"]' | base64 -d | base64
 -d
awesome%

Then I went into controller.go and poked around. Removing base64 encoding like secretDataMap[secretKey] = secretData here fixed my problem: https://github.com/itscontained/secret-manager/blob/8ea959e349444cf020f546e0da4313f193948a03/pkg/controller/externalsecret/controller.go#L179-L180

I also added logging to verify that the data wasn't already base64 encoded with:

        fmt.Printf("Secret %s = %s\n", secretKey, v)

which prints:

2020-10-12T23:48:31.070304612Z Secret yessssssssss = this is secret
2020-10-12T23:48:31.070308806Z Secret yetibot_secret = secret value
2020-10-12T23:48:31.070313366Z Secret another_one = awesome
2020-10-12T23:48:31.070317958Z Secret embedded-enabled = false
2020-10-12T23:48:31.070322966Z Secret observers-enabled = false

Running off my own docker image based on commit 999e7d0.

Ideas? 🤔

[mcavoyk]

Odd, this section should be the only logic to encode before submitting to the Kubernetes API, but anecdotally I have seen this issue when working on adding GCP SecretManager support, although it appeared that the fix there was that the Google SDK was returning the contents base64'd.

End-to-end testing suite was recently added to this project, but currently only covers AWS SecretManager, we can re-check the AWS secret manager testing is verifying the content is not double base64 encoded and add testing for vault (https://github.com/itscontained/secret-manager/issues/20).