search

ondat / trousseau

Store and access your secrets the Kubernetes native way with any external KMS.
https://trousseau.io
Apache License 2.0
178 stars 11 forks source link

[SUPPORT] Internal error occurred: unable to transform key "...": invalid padding on input #91

Closed jficz closed 2 years ago

jficz commented 2 years ago

When Trousseau is deployed in K8s cluster following the guide with Vault being hosted externally, something breaks when a new namespace is created and a secret is then created in that namespace.

Expected behavior

Secrets are accessible.

Current behavior

When accessing a secret in a new namespace via kubectl, instead of the secret an error like this returned: Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/default/testsecret2": invalid padding on input

Context

Freshly deployed cluster (using Kubespray) with Trousseau and Vault integration set up following the guide in Gitlhub project Wiki, Vault is hosted externally to the cluster.

The deployment (both at K8s side and at Vault side) is automated using Terraform, coded as close to the guide as possible.

There is nothing relevant in Trousseau pod logs. As far as I can tell, this is what appears in Trousseau pod logs when kubectl tries to access the secret:

I0518 10:52:18.201730       1 grpc.go:51] decrypt request started 
I0518 10:52:18.217137       1 grpc.go:62] decrypt request complete

It's strange that the secret created in the default namespace is accessible at first but not after the other secret in test NS is created. Also, secrets in kube-systems are apparently not affected by this. All secrets created with this method are confirmed to be in fact encrypted via Trousseau (by looking into raw etcd records directly).

This is a typical session where the error manifests (first thing after the cluster is provisioned and Trousseau integration is set up):

root@kubemaster-01 ~ # kubectl apply -f testsecret.yaml 
secret/testsecret2 created

root@kubemaster-01 ~ # kubectl -n default get secret
NAME                  TYPE                                  DATA   AGE
default-token-wz8qs   kubernetes.io/service-account-token   3      57m
testsecret2           Opaque                                1      5s

root@kubemaster-01 ~ # kubectl create namespace test
namespace/test created

root@kubemaster-01 ~ # cat testsecret-nstest.yaml 
apiVersion: v1
data:
  password: trustno1
kind: Secret
metadata:
  name: testsecret
  namespace: test
type: Opaque

root@kubemaster-01 ~ # kubectl apply -f testsecret-nstest.yaml 
secret/testsecret created

root@kubemaster-01 ~ # kubectl -n test get secret
Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/test/default-token-cb4zj": invalid padding on input

root@kubemaster-01 ~ # kubectl -n default get secret
Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets/default/testsecret2": invalid padding on input

root@kubemaster-01 ~ # cat testsecret-nskubesystem.yaml 
apiVersion: v1
data:
  password: trustno1
kind: Secret
metadata:
  name: testsecret
  namespace: kube-system
type: Opaque

root@kubemaster-01 ~ # kubectl apply -f testsecret-nskubesystem.yaml 
secret/testsecret created

root@kubemaster-01 ~ # kubectl -n kube-system get secret testsecret
NAME         TYPE     DATA   AGE
testsecret   Opaque   1      32s
romdalf commented 2 years ago

Tried to reproduce without succdess on a rke2 1.22.9+rke2r2 deployed on a CentOS 8 Stream

Containers:
  vault-kms-provider:
    Container ID:  containerd://1bc344ac1060d2b706c949f6c19f4a0410ca307f44e927f00b4866b86d42510a
    Image:         ghcr.io/ondat/trousseau:v1.1.3
    Image ID:      ghcr.io/ondat/trousseau@sha256:fb78cb70ea0ac7e601f6262fdd7763dbc02d90f0142b69b6d1f236c889eb0518
Vault v1.10.3 (af866591ee60485f05d6e32dd63dde93df686dfb)

Could you share the Terraform part so that I can deploy within the lab rig?

[root@bastion issues-91]# kubectl get nodes 
NAME             STATUS   ROLES                       AGE   VERSION
node01.sat.lab   Ready    control-plane,etcd,master   24h   v1.22.9+rke2r2
node02.sat.lab   Ready    control-plane,etcd,master   24h   v1.22.9+rke2r2
node03.sat.lab   Ready    control-plane,etcd,master   24h   v1.22.9+rke2r2

[root@bastion issues-91]# kubectl get pods -A |grep vault
kube-system   vault-kms-provider-2b9wb                                1/1     Running     5 (20h ago)      20h
kube-system   vault-kms-provider-fdd4q                                1/1     Running     0                20h
kube-system   vault-kms-provider-x897p                                1/1     Running     0                20h

[root@bastion issues-91]# cat testsecret-nsdefault.yaml 
---
apiVersion: v1
data:
  password: trustno1
kind: Secret
metadata:
  name: testsecret
type: Opaque

[root@bastion issues-91]# kubectl apply -f testsecret-nsdefault.yaml 
secret/testsecret created

[root@bastion issues-91]# kubectl delete namespace test
namespace "test" deleted

[root@bastion issues-91]# kubectl create namespace test
namespace/test created

[root@bastion issues-91]# cat testsecret-nstest.yaml 
---
apiVersion: v1
data:
  password: trustno1
kind: Secret
metadata:
  name: testsecret
  namespace: test
type: Opaque

[root@bastion issues-91]# kubectl apply -f testsecret-nstest.yaml 
secret/testsecret created

[root@bastion issues-91]# kubectl -n test get secret
NAME                  TYPE                                  DATA   AGE
default-token-6p865   kubernetes.io/service-account-token   3      27s
testsecret            Opaque                                1      10s

[root@bastion issues-91]# kubectl -n default get secret
NAME                  TYPE                                  DATA   AGE
default-token-g5n9c   kubernetes.io/service-account-token   3      24h
secret-post-deploy    Opaque                                1      20h
secret-pre-deploy     Opaque                                1      22h
secret0               Opaque                                1      20h
testsecret            Opaque                                1      54s
testsecret1           Opaque                                1      3h5m

[root@bastion issues-91]# cp testsecret-nstest.yaml testsecret-nskubesystem.yaml
[root@bastion issues-91]# vi testsecret-nskubesystem.yaml 
[root@bastion issues-91]# kubectl apply -f testsecret-nskubesystem.yaml 
secret/testsecret created

[root@bastion issues-91]# kubectl -n kube-system get secret testsecret
NAME         TYPE     DATA   AGE
testsecret   Opaque   1      8s

[root@bastion issues-91]# kubectl -n kube-system exec etcd-node01.sat.lab -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get /registry/secrets/default/testsecret" | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 74 65 73 74 73 65  |s/default/testse|
00000020  63 72 65 74 0a 6b 38 73  3a 65 6e 63 3a 6b 6d 73  |cret.k8s:enc:kms|
00000030  3a 76 31 3a 76 61 75 6c  74 70 72 6f 76 69 64 65  |:v1:vaultprovide|
00000040  72 3a 00 59 76 61 75 6c  74 3a 76 31 3a 38 34 44  |r:.Yvault:v1:84D|
00000050  32 49 4f 77 4a 79 35 44  49 4e 64 51 7a 70 77 56  |2IOwJy5DINdQzpwV|
00000060  57 57 6d 49 67 44 78 63  75 74 45 52 68 73 37 68  |WWmIgDxcutERhs7h|
00000070  33 33 6d 41 6d 65 4d 51  6f 45 71 48 6f 52 58 33  |33mAmeMQoEqHoRX3|
00000080  4f 67 50 69 4e 38 55 46  4b 68 35 58 4e 64 5a 31  |OgPiN8UFKh5XNdZ1|
00000090  42 2f 6b 7a 6d 6e 61 4e  6b 75 73 44 54 0d 2f 4c  |B/kzmnaNkusDT./L|
000000a0  da 49 9e ee 6f 8a 71 74  b1 83 21 b9 4b 02 49 2f  |.I..o.qt..!.K.I/|
000000b0  0e fc f6 80 63 79 dc 32  fa db c9 b9 44 ca 95 12  |....cy.2....D...|
000000c0  4f cf e3 15 19 2a e9 16  f9 de 8c 31 79 1f eb bd  |O....*.....1y...|
000000d0  19 55 ed 5f 21 6c 03 0e  b1 66 e0 a7 83 84 32 7f  |.U._!l...f....2.|
000000e0  93 ef fa 2a a5 08 8c 5f  bd 5b 58 8c 1c fe 3b a7  |...*..._.[X...;.|
000000f0  a7 5b 62 34 64 b9 5b 7b  07 ec df 11 cf 1a 75 34  |.[b4d.[{......u4|
00000100  b0 17 0c b6 3c b5 18 29  2d 7f 8c e3 51 0e a3 5c  |....<..)-...Q..\|
00000110  c5 33 67 21 f7 6e d0 82  bf aa 88 f9 c9 10 3e ee  |.3g!.n........>.|
00000120  0b c4 51 21 7d ad 23 6c  9c 91 f8 77 53 50 49 72  |..Q!}.#l...wSPIr|
00000130  e5 f6 ab 01 f4 9c 6f 60  2b 0b 03 e6 3b 8e f7 54  |......o`+...;..T|
00000140  23 01 ff 47 9d a6 94 b0  11 61 35 f1 17 5e 18 7c  |#..G.....a5..^.||
00000150  07 aa b0 f4 6c 53 79 ef  e5 54 33 dc a5 1c bc 5e  |....lSy..T3....^|
00000160  a4 65 a9 67 a7 05 fe 26  97 58 c9 ab 07 1a d5 aa  |.e.g...&.X......|
00000170  85 61 9c 14 81 3b 3e 10  29 7a e1 16 4a d2 ae 5d  |.a...;>.)z..J..]|
00000180  a4 a8 d4 bd d3 21 a3 01  8b 26 28 c0 35 f4 d7 3e  |.....!...&(.5..>|
00000190  95 7a 22 8c 2d ea 91 03  78 c0 d9 de a2 11 24 1d  |.z".-...x.....$.|
000001a0  82 79 c8 ac 59 13 6e 05  39 53 26 bd fa 87 19 ff  |.y..Y.n.9S&.....|
000001b0  8a f2 f1 0e 31 26 ee a1  36 3b 5c 2b 96 20 24 32  |....1&..6;\+. $2|
000001c0  e8 61 0c 8c 42 6b 13 b0  e1 71 2a cb 77 09 6d df  |.a..Bk...q*.w.m.|
000001d0  59 2c e6 de e0 48 f8 92  23 ae a2 7e dc ac 8f 1b  |Y,...H..#..~....|
000001e0  0b 05 65 5f 7f be 3c d3  84 27 e7 6a 8d 2f 02 14  |..e_..<..'.j./..|
000001f0  75 76 33 70 76 8c b0 b3  c3 f6 eb b8 3b 56 c4 8e  |uv3pv.......;V..|
00000200  fd bc 3f 00 da 3a a4 8f  b4 51 89 cf 32 6e 94 dd  |..?..:...Q..2n..|
00000210  65 2c 4d 74 c5 bb c7 02  d8 8f 1e a3 27 fd 84 75  |e,Mt........'..u|
00000220  3c dc c6 db 95 36 e7 7e  66 2e 31 9f fb fe 9d 8b  |<....6.~f.1.....|
00000230  e1 f2 a6 e8 42 fb ad 1b  af 4c 4a 85 27 3d a0 c4  |....B....LJ.'=..|
00000240  d8 78 f3 6d d6 43 ab e5  82 54 70 a6 de 99 37 67  |.x.m.C...Tp...7g|
00000250  6d b2 3b b5 40 e4 b1 b5  a7 10 db 55 af 79 09 72  |m.;.@......U.y.r|
00000260  8b 20 e3 53 22 da ad fa  5a da 48 35 da a4 32 5a  |. .S"...Z.H5..2Z|
00000270  93 2f da 5a 3f cc 2e 17  f4 fe 8e 49 e9 2f 7f 4c  |./.Z?......I./.L|
00000280  f1 34 ea dc 6a fb e9 5e  f9 e0 13 05 bb a9 2d 04  |.4..j..^......-.|
00000290  fd 28 ec e9 7c 2b 27 2e  7d 23 c3 0c 70 40 57 90  |.(..|+'.}#..p@W.|
000002a0  67 4d 43 74 5c bf 20 16  2e d4 f0 72 3b 8f 3f db  |gMCt\. ....r;.?.|
000002b0  f2 42 7a 22 9f c2 e0 19  a9 7d c3 d3 00 e0 2b 1e  |.Bz".....}....+.|
000002c0  dd 71 91 a4 59 35 6f ee  e8 79 de 88 e7 26 e4 c3  |.q..Y5o..y...&..|
000002d0  e8 51 af ff 99 76 bc e7  02 5b 67 df 03 6d 0a 7d  |.Q...v...[g..m.}|
000002e0  ce f2 98 78 67 44 93 bb  69 52 97 0d d6 0a        |...xgD..iR....|
000002ee

[root@bastion issues-91]# kubectl -n kube-system exec etcd-node01.sat.lab -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get /registry/secrets/test/testsecret" | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 74 65 73 74 2f 74  65 73 74 73 65 63 72 65  |s/test/testsecre|
00000020  74 0a 6b 38 73 3a 65 6e  63 3a 6b 6d 73 3a 76 31  |t.k8s:enc:kms:v1|
00000030  3a 76 61 75 6c 74 70 72  6f 76 69 64 65 72 3a 00  |:vaultprovider:.|
00000040  59 76 61 75 6c 74 3a 76  31 3a 68 58 75 31 56 41  |Yvault:v1:hXu1VA|
00000050  7a 4c 62 61 34 78 55 74  70 59 79 6a 48 59 4d 73  |zLba4xUtpYyjHYMs|
00000060  56 2b 56 71 65 67 73 73  47 78 66 4e 32 67 58 42  |V+VqegssGxfN2gXB|
00000070  5a 37 56 33 62 6e 34 74  4c 61 55 58 4e 64 4a 6d  |Z7V3bn4tLaUXNdJm|
00000080  37 65 47 58 79 63 6c 6d  76 5a 6a 50 37 66 38 2f  |7eGXyclmvZjP7f8/|
00000090  6a 33 6c 41 36 68 52 42  6b 74 20 90 ad ce 6d 11  |j3lA6hRBkt ...m.|
000000a0  96 50 79 e8 96 08 e9 be  39 bf 01 c6 19 0b ed 87  |.Py.....9.......|
000000b0  5d 68 fc ed 58 64 d9 b5  91 e9 8d cc b0 2b 55 74  |]h..Xd.......+Ut|
000000c0  0b 61 d5 d9 72 8a 9e 6d  cf 5d d9 17 e4 6b ab 16  |.a..r..m.]...k..|
000000d0  57 3f 99 85 c7 ae 73 57  37 b0 e7 ca 40 08 99 ae  |W?....sW7...@...|
000000e0  d0 41 bd d3 88 ff 4c 63  4b 9a 2f da 11 45 98 e3  |.A....LcK./..E..|
000000f0  aa 2b a4 e2 a4 86 b2 59  e7 d9 4c 38 9b e5 3e 32  |.+.....Y..L8..>2|
00000100  93 bd ec 9d 25 a0 30 b3  c2 53 88 00 5e a7 ac 4e  |....%.0..S..^..N|
00000110  ea e4 98 dc b7 16 51 1b  ca 8a 34 39 5c 66 dc 8d  |......Q...49\f..|
00000120  39 bb 18 ae 52 e7 23 24  ca cb 49 1e 78 8c 21 be  |9...R.#$..I.x.!.|
00000130  10 13 29 de 33 e9 05 8b  11 b4 17 5e 59 c3 de e4  |..).3......^Y...|
00000140  af 4a 4b 60 de 27 97 a2  d9 3d cb 7f af cc 6e 6e  |.JK`.'...=....nn|
00000150  13 8c 95 e2 fb a7 d4 60  e9 49 0a 0c a9 54 eb 89  |.......`.I...T..|
00000160  12 43 c7 7d 77 bf 04 2b  79 70 64 82 16 f1 93 8e  |.C.}w..+ypd.....|
00000170  22 d8 ec f7 6f 27 20 0c  37 af ec 3f ce c8 35 b8  |"...o' .7..?..5.|
00000180  18 4a f6 ed 0c 42 c2 61  2a 32 a0 b4 a3 19 d1 4f  |.J...B.a*2.....O|
00000190  18 fd 14 f8 23 7c 68 10  7e f8 5b 68 bc a8 e7 81  |....#|h.~.[h....|
000001a0  37 cb ef f4 2a 3c 84 9c  6c 6b 17 ac 51 56 13 41  |7...*<..lk..QV.A|
000001b0  c1 5d d4 b2 fe e4 c8 ca  0a be 74 45 46 83 c7 c6  |.]........tEF...|
000001c0  42 ce ad d5 6b 8b 78 65  e7 9c 82 ef d0 91 6b 88  |B...k.xe......k.|
000001d0  35 02 a4 85 30 91 0d a1  1e ae b4 1a f1 86 02 ed  |5...0...........|
000001e0  3f 16 b8 68 de da 27 98  75 16 e1 2f 6b de 79 cc  |?..h..'.u../k.y.|
000001f0  11 22 ba 35 dc 11 ee 4c  40 50 05 54 aa 76 db 63  |.".5...L@P.T.v.c|
00000200  9e 69 7a 2e b8 d4 21 64  c7 96 09 c1 af 2a 66 85  |.iz...!d.....*f.|
00000210  59 c7 c8 fc e9 c9 39 ba  0e 4e a6 34 3c a2 5a 03  |Y.....9..N.4<.Z.|
00000220  07 73 3c b0 4a 66 3d 43  50 4b 2a 1d f3 52 06 73  |.s<.Jf=CPK*..R.s|
00000230  fc a3 b3 0d 96 e6 7b b7  3a e1 c9 11 f8 56 a4 f2  |......{.:....V..|
00000240  2d 6c 57 cf 93 bc 68 ed  ee 85 7c e9 17 89 f3 bb  |-lW...h...|.....|
00000250  42 5a b2 14 72 93 c3 7b  3c ad b2 31 ec 73 6f 73  |BZ..r..{<..1.sos|
00000260  98 0f f0 2f d9 d2 cc b1  71 5f 8a 3b b1 7c d5 4f  |.../....q_.;.|.O|
00000270  0f a0 97 83 44 6c e0 e0  81 5f 9b a7 eb 87 9f ac  |....Dl..._......|
00000280  74 1f a4 cd b3 83 c7 0a  17 ae 46 e8 6b 94 7a b6  |t.........F.k.z.|
00000290  6f 8f c6 68 5b 80 3e 40  39 b8 28 4b e7 65 39 73  |o..h[.>@9.(K.e9s|
000002a0  b6 1b 52 c1 3f 6a 69 0c  97 88 b4 e9 78 29 a3 3f  |..R.?ji.....x).?|
000002b0  8a 5f 78 61 d3 88 a0 e7  b8 b0 67 0d a2 c1 8f 3c  |._xa......g....<|
000002c0  6f b4 5a 3e 9b 6d 64 2b  b2 d0 ff fc 75 4c 6e fe  |o.Z>.md+....uLn.|
000002d0  af a8 1f 98 39 14 6e 16  4b 44 0a                 |....9.n.KD.|
000002db

[root@bastion issues-91]# kubectl -n kube-system exec etcd-node01.sat.lab -- sh -c "ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get /registry/secrets/kube-system/testsecret" | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 6b 75 62 65 2d 73  79 73 74 65 6d 2f 74 65  |s/kube-system/te|
00000020  73 74 73 65 63 72 65 74  0a 6b 38 73 3a 65 6e 63  |stsecret.k8s:enc|
00000030  3a 6b 6d 73 3a 76 31 3a  76 61 75 6c 74 70 72 6f  |:kms:v1:vaultpro|
00000040  76 69 64 65 72 3a 00 59  76 61 75 6c 74 3a 76 31  |vider:.Yvault:v1|
00000050  3a 69 33 59 53 38 75 48  75 72 43 6a 45 64 4f 62  |:i3YS8uHurCjEdOb|
00000060  70 70 73 38 53 76 2b 6a  32 75 61 4a 58 72 48 55  |pps8Sv+j2uaJXrHU|
00000070  56 4d 30 4b 62 6e 54 6f  4f 4e 52 6e 62 51 4f 72  |VM0KbnToONRnbQOr|
00000080  75 2b 72 51 33 47 79 6c  77 71 6a 65 35 76 77 4e  |u+rQ3Gylwqje5vwN|
00000090  37 78 50 46 61 58 5a 6a  4f 6a 43 6c 75 79 4d 72  |7xPFaXZjOjCluyMr|
000000a0  64 89 e5 52 91 bd 64 7d  a8 3f b8 d7 68 48 52 36  |d..R..d}.?..hHR6|
000000b0  57 19 4f 1f e2 e7 22 e1  14 e1 78 81 8f f0 4e 41  |W.O..."...x...NA|
000000c0  35 50 ba 43 3f e0 9b 43  f4 a6 d2 fb eb 00 36 b1  |5P.C?..C......6.|
000000d0  37 b7 c8 fc e4 d4 77 ae  a8 1f bd af 16 0c c9 f4  |7.....w.........|
000000e0  ac e6 f2 4d 34 15 08 1e  70 61 a9 73 0a 00 df 25  |...M4...pa.s...%|
000000f0  a9 70 5e f0 7d 2c 48 20  99 2d 20 8f 0f fd 41 37  |.p^.},H .- ...A7|
00000100  aa 16 c4 97 80 4a 39 bc  1a a4 bd ab a9 c0 de 98  |.....J9.........|
00000110  74 c8 24 df 18 71 7a 90  b3 b8 e3 32 dc b2 bb 8d  |t.$..qz....2....|
00000120  38 44 17 a5 4d 4c e5 c6  27 a8 77 8e 0b cc d5 30  |8D..ML..'.w....0|
00000130  1a a8 b3 92 6d a1 f2 a9  52 58 e0 34 3f b3 5b 62  |....m...RX.4?.[b|
00000140  c1 a6 b5 cf 22 0d 10 99  45 d3 1a bb b0 71 ce 75  |...."...E....q.u|
00000150  5a 29 50 f8 49 61 7f 58  e4 5e 02 fc 46 05 06 82  |Z)P.Ia.X.^..F...|
00000160  d8 06 7e f1 34 8f ad ef  e8 97 82 5e d8 bf 7d 6e  |..~.4......^..}n|
00000170  50 29 0a bc 15 1b 49 32  22 79 df bc a6 f3 87 da  |P)....I2"y......|
00000180  3d 35 4b c6 36 32 54 94  99 46 d4 b6 51 c1 a3 fe  |=5K.62T..F..Q...|
00000190  e9 51 4c eb 4a 4f c8 3c  3c 00 7c 4a d9 03 c9 90  |.QL.JO.<<.|J....|
000001a0  84 21 ee 97 ee 32 eb 9c  75 de ca e3 2f 1d fc fd  |.!...2..u.../...|
000001b0  f6 b6 b7 07 26 90 c4 61  83 cc 9c 70 e0 bc 30 e2  |....&..a...p..0.|
000001c0  af e1 d4 02 a7 cc 26 73  ce 0c 9e aa b7 66 28 97  |......&s.....f(.|
000001d0  76 e9 03 8c a1 e9 fc 1e  11 5c 61 14 9b 1f bc 43  |v........\a....C|
000001e0  aa 6c 8e 16 ce 7c f1 2c  63 a5 59 b3 63 d1 8b c0  |.l...|.,c.Y.c...|
000001f0  8f 77 e9 8c 23 96 9d 8e  2a 6e 7e 1a d1 6e 77 5f  |.w..#...*n~..nw_|
00000200  b2 66 f7 36 3b d5 bd 0e  4c ba f9 ab 56 01 22 4c  |.f.6;...L...V."L|
00000210  e8 a8 d7 d9 39 8b 41 a9  d3 df 90 6f 58 f7 3a 8b  |....9.A....oX.:.|
00000220  01 db ec 91 e3 f5 33 c8  10 4c 7c 29 ab 3f cd 82  |......3..L|).?..|
00000230  d2 5f 4c ce 42 12 e5 ad  aa 30 c2 ea 96 db 67 5b  |._L.B....0....g[|
00000240  d3 f2 84 65 6e ae 47 6f  4f 0e 0c 05 93 2f 92 b4  |...en.GoO..../..|
00000250  23 91 38 30 e6 00 45 5a  e4 2f de 2c bf 0d f5 23  |#.80..EZ./.,...#|
00000260  b7 58 bf 0e f5 33 7e 0c  fa b0 d7 b4 56 07 dd d3  |.X...3~.....V...|
00000270  bb 6d 7e 58 be 3e b2 cc  c1 92 b4 6b 6b 28 c5 8e  |.m~X.>.....kk(..|
00000280  c4 49 57 e5 c2 5f e6 1c  3c 6b 7f be 0b f2 1b 6e  |.IW.._..<k.....n|
00000290  42 67 24 33 c6 6c a1 a9  4f 1b 51 4d 63 c5 ba f9  |Bg$3.l..O.QMc...|
000002a0  a5 a6 7f 50 83 61 7d e2  30 83 9b b9 67 40 66 b4  |...P.a}.0...g@f.|
000002b0  35 35 ff d7 c6 1f 6c 71  47 a1 f8 f4 4e d1 7a bc  |55....lqG...N.z.|
000002c0  0e aa e9 b6 f4 c6 d1 d0  36 f7 78 43 db 92 b0 cc  |........6.xC....|
000002d0  d9 6f 8e c2 b7 da 27 c2  2f 19 39 1d 39 fd cf c9  |.o....'./.9.9...|
000002e0  44 a5 e9 81 44 39 59 55  3f 83 fb f6 b1 dd 32 cd  |D...D9YU?.....2.|
000002f0  12 0a                                             |..|
000002f2
[root@bastion issues-91]# 

[root@bastion issues-91]# kubectl logs -n kube-system vault-kms-provider-2b9wb
Defaulted container "vault-kms-provider" out of: vault-kms-provider, vault-agent (init)
{"level":"info","timestamp":"2022-06-08T21:44:27.186138574Z","caller":"kubernetes-kms-vault/main.go:74","msg":"Starting VaultEncryptionServiceServer service","version":"dev","buildDate":""}
{"level":"Level(-2)","timestamp":"2022-06-08T21:44:27.186278092Z","caller":"config/config.go:20","msg":"Populating AppConfig...","path":"/opt/trousseau/config.yaml"}
{"level":"info","timestamp":"2022-06-08T21:44:27.187258147Z","caller":"metrics/exporter.go:17","msg":"Metrics backend","exporter":"prometheus"}
{"level":"info","timestamp":"2022-06-08T21:44:27.187408735Z","caller":"metrics/prometheus_exporter.go:26","msg":"Prometheus metrics server starting","address":"8095"}
{"level":"info","timestamp":"2022-06-08T21:44:27.187532377Z","caller":"kubernetes-kms-vault/main.go:107","msg":"Listening for connections","address":"/opt/vault-kms/vaultkms.socket"}
{"level":"Level(-3)","timestamp":"2022-06-08T21:44:27.187621861Z","caller":"server/health.go:33","msg":"Initialize health check\n"}
[root@bastion issues-91]# kubectl logs -n kube-system vault-kms-provider-fdd4q
Defaulted container "vault-kms-provider" out of: vault-kms-provider, vault-agent (init)
{"level":"info","timestamp":"2022-06-08T21:55:35.581700161Z","caller":"kubernetes-kms-vault/main.go:74","msg":"Starting VaultEncryptionServiceServer service","version":"dev","buildDate":""}
{"level":"Level(-2)","timestamp":"2022-06-08T21:55:35.581820347Z","caller":"config/config.go:20","msg":"Populating AppConfig...","path":"/opt/trousseau/config.yaml"}
{"level":"info","timestamp":"2022-06-08T21:55:35.581915593Z","caller":"metrics/exporter.go:17","msg":"Metrics backend","exporter":"prometheus"}
{"level":"info","timestamp":"2022-06-08T21:55:35.582045288Z","caller":"metrics/prometheus_exporter.go:26","msg":"Prometheus metrics server starting","address":"8095"}
{"level":"info","timestamp":"2022-06-08T21:55:35.582382898Z","caller":"kubernetes-kms-vault/main.go:107","msg":"Listening for connections","address":"/opt/vault-kms/vaultkms.socket"}
{"level":"Level(-3)","timestamp":"2022-06-08T21:55:35.582445646Z","caller":"server/health.go:33","msg":"Initialize health check\n"}
[root@bastion issues-91]# kubectl logs -n kube-system vault-kms-provider-x897p
Defaulted container "vault-kms-provider" out of: vault-kms-provider, vault-agent (init)
{"level":"info","timestamp":"2022-06-08T21:40:01.73925563Z","caller":"kubernetes-kms-vault/main.go:74","msg":"Starting VaultEncryptionServiceServer service","version":"dev","buildDate":""}
{"level":"info","timestamp":"2022-06-08T21:40:01.739375492Z","caller":"metrics/exporter.go:17","msg":"Metrics backend","exporter":"prometheus"}
{"level":"Level(-2)","timestamp":"2022-06-08T21:40:01.739388506Z","caller":"config/config.go:20","msg":"Populating AppConfig...","path":"/opt/trousseau/config.yaml"}
{"level":"info","timestamp":"2022-06-08T21:40:01.73953721Z","caller":"metrics/prometheus_exporter.go:26","msg":"Prometheus metrics server starting","address":"8095"}
{"level":"info","timestamp":"2022-06-08T21:40:01.739973234Z","caller":"kubernetes-kms-vault/main.go:107","msg":"Listening for connections","address":"/opt/vault-kms/vaultkms.socket"}
{"level":"Level(-3)","timestamp":"2022-06-08T21:40:01.740036542Z","caller":"server/health.go:33","msg":"Initialize health check\n"}
romdalf commented 2 years ago

@jficz any luck in cheking the above?

jficz commented 2 years ago

@rovandep not yet, I'll hopefully get to it this week. A point to notice though: you're using RKE, we use "vanilla" K8s with Kubespray

romdalf commented 2 years ago

let me have a look at a kubespray testing then

romdalf commented 2 years ago

Hey, I am not successful to reproduce this issue. I tried with with kubespray, minikube, kind and rancher within no issues and we have testing going up to 1.23 for v1 and 1.24 for v2.

Question: are you able to get the data payload out of etcd? (like shown within my example).

jficz commented 2 years ago

hey, I basically recreated the whole configuration from scratch. I no longer have this specific issue but I still have issues.

The trousseau pod starts successfully but then it doesn't do any encryption operation (secrets are still created unencrypted even though API is configured to use trousseau via socket). In about ten minutes the pod fails: https://gist.github.com/jficz/5ed5b151d590d66db2b8f7d1036ffb29

Something with 403 and healthchecks but I don't really know what to make of it.

Need to add that we had issues with trousseau health checks before, too.

The 403 doesn't make much sense to me either, the token Trousseau uses has this policy attached:

path "transit/*" {
   capabilities = [ "list", "read", "update" ]
}
romdalf commented 2 years ago

could you share the vault logs?

jficz commented 2 years ago

Not much there, just a bunch of these

{
  "time": "2022-08-02T15:51:30.770308838Z",
  "type": "request",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "c787e164-9795-41ac-c024-c22d640fd0da",
    "operation": "update",
    "mount_type": "transit",
    "client_token": "hmac-sha256:<removed>",
    "namespace": {
      "id": "root"
    },
    "path": "transit/encrypt/kube-ktest-kms",
    "data": {
      "plaintext": "hmac-sha256:<removed3>"
    },
    "remote_address": "192.168.1.6",
    "remote_port": 58180
  },
  "error": "permission denied"
}
{
  "time": "2022-08-02T15:51:30.77050389Z",
  "type": "response",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "c787e164-9795-41ac-c024-c22d640fd0da",
    "operation": "update",
    "mount_type": "transit",
    "client_token": "hmac-sha256:<removed>",
    "namespace": {
      "id": "root"
    },
    "path": "transit/encrypt/kube-ktest-kms",
    "data": {
      "plaintext": "hmac-sha256:<removed3>"
    },
    "remote_address": "192.168.1.6",
    "remote_port": 58180
  },
  "response": {
    "mount_type": "transit",
    "data": {
      "error": "hmac-sha256:<removed2>"
    }
  },
  "error": "1 error occurred:\n\t* permission denied\n\n"
}
romdalf commented 2 years ago

Is Vault running on Kubernetes? If yes, same cluster as Trousseau?

Trousseau seems to have a trouble to reach Vault - the health process checks connectivity to Vault and if failed 3 times, we will terminate Trousseau.

So from a workflow perspective, here how it works:

From a doc perspective, the transit key engine is addressed by creating a policy for the dedicate token and the token itself.

So either the policy is not or wrongly set: 

vault policy write trousseau-transit-ro -<<EOF
path "transit/encrypt/trousseau-kms-vault" {
   capabilities = [ "update" ]
}
path "transit/decrypt/trousseau-kms-vault" {
   capabilities = [ "update" ]
}
EOF

or used for the dedicated token creation:

vault token create -policy=trousseau-transit-ro
Key                  Value
---                  -----
token                hvs.CAESILoUyuj8STPYKR4AGhaCJylJbkOkmlXlU8pZukoQKc_bGh4KHGh2cy5vQkpnc2g0RVNFZEpsWTA0SWlSNDBxWDQ
token_accessor       BBTat50bsupNqAQNLTXXRhr7
token_duration       768h
token_renewable      true
token_policies       ["default" "trousseau-transit-ro"]
identity_policies    []
policies             ["default" "trousseau-transit-ro"]

or the dedicated token was not or wrongly exported: 

export TROUSSEAU_TOKEN="hvs.CAESILoUyuj8STPYKR4AGhaCJylJbkOkmlXlU8pZukoQKc_bGh4KHGh2cy5vQkpnc2g0RVNFZEpsWTA0SWlSNDBxWDQ"

then the following command would not get the appropriate vaulttoken value 

vault kv put /secret/trousseau/config transitkeyname=trousseau-kms-vault \
vaultaddress=$VAULT_ADDR vaulttoken=$TROUSSEAU_TOKEN \
 ttl=30s

Let me know if it makes sense.

jficz commented 2 years ago

Turns out this was a Terraform issue in the end. The Token policy was not assigned correctly to the key. Sorry for the waves.

So far it seems even the original issue is gone and Trousseau works for me in v1.1.3 (kubespray 2.19).

Thanks!

romdalf commented 2 years ago

No waves no fun! Thanks for closing the issue and don't hesitate to share your experience in 3 months time ;)