primeroz
commented
4 years ago Reference discussion in slack
Open andreygolev opened 4 years ago
primeroz
commented
4 years ago I have been trying to reproduce this using kind both with 0.5.1 and master but so far i was not able to reproduce the issue.
@andreygolev I remember you told me this was a build of your own of the bank-vaults container but i noticed something while running in my test
in your case the reflrector , when is not able to connect upstream , prints out
E0903 14:51:31.325023 1 reflector.go:134] pkg/mod/k8s.io/client-go@v2.0.0-alpha.0.0.20181213151034-8d9ed539ba31+incompatible/tools/cache/reflector.go:95: Failed to list *v1.StatefulSet: Get https://10.96.0.1:443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refusedwhile in my case using banzaicloud/vault-operator:master
E0905 07:30:48.514570 1 reflector.go:134] pkg/mod/k8s.io/client-go@v10.0.0+incompatible/tools/cache/reflector.go:95: Failed to list *v1.StatefulSet: Get https://10.96.0.1:443/apis/apps/v1/namespaces/vault/statefulsets?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refusedsee the different version of library ?
pkg/mod/k8s.io/client-go@v2.0.0-alpha.0.0.20181213151034-8d9ed539ba31 vs pkg/mod/k8s.io/client-go@v10.0.0
Do you mind trying to use banzaicloud/vault-operator:0.6.0-rc.1 or banzaicloud/vault-operator:master and see if you still have the issue ?
they both should have support for the postgresql you need
doggvalera
commented
4 years ago @primeroz
Hi. I'm working with Andrey. We are using a master image, but the image was created 4-5 days ago. I saw that the last update of the master was a few hours ago. I will try a new image.
andreygolev
commented
4 years ago At first time it seemed to me that banzaicloud/vault-operator:master fixes the issue, but I tried to keep killing apiserver, and after 6th attempt vault pods disappeared. Maybe try pkill -9 apiserver. The mighty 9 maybe works better to reproduce the case :)
primeroz
commented
4 years ago Can you check in the logs what client-go version is using and confirm is same one as mine ?
Yeah i did pkill -9 a lot in a loop never allowing the apiserver to come back for more then 1 minute.
I will give a try to minikube since kind was easier
andreygolev
commented
4 years ago Yup, it's same.
E0905 09:39:29.875078 1 reflector.go:134] pkg/mod/k8s.io/client-go@v10.0.0+incompatible/tools/cache/reflector.go:95: Failed to list *v1.ConfigMap: Get https://10.96.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refusedVideo for the issue: https://youtu.be/tqggFjWCe08
At around 7:25 pods will be recreated
primeroz
commented
4 years ago For reference, i have been testing exactly the same behaviour as the video and so far i can't get the issue to happen
I killed and waited at least 10 times :)
still the original vault-0 is running
NAME READY STATUS RESTARTS AGE
vault-0 3/3 Running 0 125m
vault-configurer-76c55cddb-pc8ft 1/1 Running 0 125m
vault-operator-54465b7585-dbbp5 1/1 Running 0 131mThe only strange thing i can see is that whenever the apiserver is killed and come back the
generation , resourceVersion for the VAULT CRD change (increase) but the CRD UID never did change so the whole ownership of the underlying resources is not affected
Generation: 17
ResourceVersion: "13442"
UID: "a98fb42b-cffb-479a-9c3a-c12f11fdf610"This must be a very edge case!
andreygolev
commented
4 years ago Are you testing on the same minikube and kubernetes versions?
k get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready master 2d v1.15.2 192.168.64.5 <none> Buildroot 2018.05.3 4.15.0 docker://18.9.8minikube version: v1.3.1
primeroz
commented
4 years ago yes
minikube start --vm-driver=kvm2 --wait=true --cpus 2 --memory 4096 --kubernetes-version=v1.15.2
andreygolev
commented
4 years ago I just tried to wipe whole minikube, recreate all from scratch and problem still reproduces.
I attached all the manifests that I literally apply to a fresh minikube.
Then do same pkill -9 apiserver.
The thing is that I don't wait more than 1 minute between apiserver kills after it came up.
Maybe that will also help.
operator-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: vault-operator
rules:
- apiGroups:
- ""
resources:
- pods
- services
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- list
- get
- create
- update
- watch
- apiGroups:
- apps
resources:
- replicasets
verbs:
- list
- get
- apiGroups:
- apps
resources:
- deployments
- statefulsets
verbs:
- '*'
- apiGroups:
- monitoring.coreos.com
resources:
- servicemonitors
verbs:
- update
- list
- get
- create
- apiGroups:
- vault.banzaicloud.com
resources:
- '*'
verbs:
- '*'
- apiGroups:
- etcd.database.coreos.com
resources:
- "*"
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: vault-operator
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- watch
- apiGroups: [""]
resources: ["secrets"]
verbs:
- create
- update
- get
- list
- watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-operator
subjects:
- kind: ServiceAccount
name: vault-operator
namespace: default
roleRef:
kind: Role
name: vault-operator
apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-operator
subjects:
- kind: ServiceAccount
name: vault-operator
namespace: default
roleRef:
kind: ClusterRole
name: vault-operator
apiGroup: rbac.authorization.k8s.iorbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: vault
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-secrets
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- "*"
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: vault-secrets
roleRef:
kind: Role
name: vault-secrets
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: vault
---
# This binding allows the deployed Vault instance to authenticate clients
# through Kubernetes ServiceAccounts (if configured so).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault
namespace: defaultoperator.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: vaults.vault.banzaicloud.com
spec:
group: vault.banzaicloud.com
names:
kind: Vault
listKind: VaultList
plural: vaults
singular: vault
scope: Namespaced
version: v1alpha1
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8383"
name: vault-operator
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
name: vault-operator
template:
metadata:
labels:
name: vault-operator
spec:
serviceAccountName: vault-operator
containers:
- name: vault-operator
image: banzaicloud/vault-operator:master
imagePullPolicy: IfNotPresent
# args:
# - -verbose
# - -sync_period=10s
ports:
- containerPort: 8383
name: metrics
command:
- vault-operator
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 4
periodSeconds: 10
failureThreshold: 1
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 4
periodSeconds: 10
failureThreshold: 1
env:
- name: WATCH_NAMESPACE
# Use this to watch all namespaces
value: ""
# Use this to watch own namespace only
# valueFrom:
# fieldRef:
# fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "vault-operator"
resources:
limits:
cpu: "100m"
memory: "128Mi"cr.yaml
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
name: "vault"
spec:
size: 1
image: vault:1.1.0
bankVaultsImage: banzaicloud/bank-vaults:master
# Common annotations for all created resources
annotations:
common/annotation: "true"
# Vault Pods , Services and TLS Secret annotations
vaultAnnotations:
type/instance: "vault"
# Vault Configurer Pods and Services annotations
vaultConfigurerAnnotations:
type/instance: "vaultconfigurer"
# Vault Pods , Services and TLS Secret labels
vaultLabels:
example.com/log-format: "json"
# Vault Configurer Pods and Services labels
vaultConfigurerLabels:
example.com/log-format: "string"
# Support for nodeAffinity Rules
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key : "node-role.kubernetes.io/your_role"
# operator: In
# values: ["true"]
# Support for pod nodeSelector rules to control which nodes can be chosen to run
# the given pods
# nodeSelector:
# "node-role.kubernetes.io/your_role": "true"
# Support for node tolerations that work together with node taints to control
# the pods that can like on a node
# tolerations:
# - effect: NoSchedule
# key: node-role.kubernetes.io/your_role
# operator: Equal
# value: "true"
# Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
serviceAccount: vault
# Specify the Service's type where the Vault Service is exposed
# Please note that some Ingress controllers like https://github.com/kubernetes/ingress-gce
# forces you to expose your Service on a NodePort
serviceType: ClusterIP
# Request an Ingress controller with the default configuration
ingress:
# Specify Ingress object annotations here, if TLS is enabled (which is by default)
# the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations
# to support TLS backends
annotations:
# Override the default Ingress specification here
# This follows the same format as the standard Kubernetes Ingress
# See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#ingressspec-v1beta1-extensions
spec: {}
# Use local disk to store Vault file data, see config section.
volumes:
- name: vault-file
persistentVolumeClaim:
claimName: vault-file
volumeMounts:
- name: vault-file
mountPath: /vault/file
# Support for distributing the generated CA certificate Secret to other namespaces.
# Define a list of namespaces or use ["*"] for all namespaces.
caNamespaces:
- "*"
# Describe where you would like to store the Vault unseal keys and root token.
unsealConfig:
options:
# The preFlightChecks flag enables unseal and root token storage tests
# This is true by default
preFlightChecks: true
kubernetes:
secretNamespace: default
# A YAML representation of a final vault config file.
# See https://www.vaultproject.io/docs/configuration/ for more information.
config:
storage:
file:
path: "${ .Env.VAULT_STORAGE_FILE }" # An example how Vault config environment interpolation can be used
listener:
tcp:
address: "0.0.0.0:8200"
# Uncommenting the following line and deleting tls_cert_file and tls_key_file disables TLS
# tls_disable: true
tls_cert_file: /vault/tls/server.crt
tls_key_file: /vault/tls/server.key
telemetry:
statsd_address: localhost:9125
ui: true
# See: https://github.com/banzaicloud/bank-vaults#example-external-vault-configuration for more details.
externalConfig:
policies:
- name: allow_secrets
rules: path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
auth:
- type: kubernetes
roles:
# Allow every pod in the default namespace to use the secret kv store
- name: default
bound_service_account_names: ["default", "vault-secrets-webhook"]
bound_service_account_namespaces: ["default", "vswh"]
policies: allow_secrets
ttl: 1h
secrets:
- path: secret
type: kv
description: General secrets.
options:
version: 2
# Allows writing some secrets to Vault (useful for development purposes).
# See https://www.vaultproject.io/docs/secrets/kv/index.html for more information.
startupSecrets:
- type: kv
path: secret/data/accounts/aws
data:
data:
AWS_ACCESS_KEY_ID: secretId
AWS_SECRET_ACCESS_KEY: s3cr3t
- type: kv
path: secret/data/dockerrepo
data:
data:
DOCKER_REPO_USER: dockerrepouser
DOCKER_REPO_PASSWORD: dockerrepopassword
- type: kv
path: secret/data/mysql
data:
data:
MYSQL_ROOT_PASSWORD: s3cr3t
vaultEnvsConfig:
- name: VAULT_LOG_LEVEL
value: debug
- name: VAULT_STORAGE_FILE
value: "/vault/file"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: vault-file
spec:
# https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class-1
# storageClassName: ""
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# ---
# apiVersion: v1
# kind: PersistentVolume
# metadata:
# name: vault-file
# spec:
# capacity:
# storage: 1Gi
# accessModes:
# - ReadWriteOnce
# persistentVolumeReclaimPolicy: Recycle
# hostPath:
# path: /vault/fileWe have some updates there. So, we have vault-operator scaled to 0. And guess what? All resources that were created by vault-operator gone even without vault-operator!
primeroz
commented
4 years ago Did you see anything in the apiserver / controller-manager logs ?
I tried this but once i scaled the operator to 0 , and waited at least 4 minutes by now , nothing happened to the resources
kubectl get all -n vault
NAME READY STATUS RESTARTS AGE
pod/vault-0 3/3 Running 0 6m3s
pod/vault-configurer-7b8f4fd595-ktjsg 1/1 Running 0 6m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/vault ClusterIP 10.97.34.62 <none> 8200/TCP,8201/TCP,9091/TCP,9102/TCP 6m3s
service/vault-0 ClusterIP 10.100.49.7 <none> 8200/TCP,8201/TCP,9091/TCP 6m3s
service/vault-configurer ClusterIP 10.105.152.233 <none> 9091/TCP 6m3s
service/vault-operator-metrics ClusterIP 10.110.71.98 <none> 8383/TCP 6m5s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/vault-configurer 1/1 1 1 6m3s
deployment.apps/vault-operator 0/0 0 0 6m20s
NAME DESIRED CURRENT READY AGE
replicaset.apps/vault-configurer-7b8f4fd595 1 1 1 6m3s
replicaset.apps/vault-operator-59677fddb4 0 0 0 6m20s
NAME READY AGE
statefulset.apps/vault 1/1 6m3sScaled the operator back up to 1 , after 2 minutes still all good
kubectl get all -n vault
NAME READY STATUS RESTARTS AGE
pod/vault-0 3/3 Running 0 8m25s
pod/vault-configurer-7b8f4fd595-ktjsg 1/1 Running 0 8m25s
pod/vault-operator-59677fddb4-ftnqh 1/1 Running 0 2m4s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/vault ClusterIP 10.97.34.62 <none> 8200/TCP,8201/TCP,9091/TCP,9102/TCP 8m25s
service/vault-0 ClusterIP 10.100.49.7 <none> 8200/TCP,8201/TCP,9091/TCP 8m25s
service/vault-configurer ClusterIP 10.105.152.233 <none> 9091/TCP 8m25s
service/vault-operator-metrics ClusterIP 10.110.71.98 <none> 8383/TCP 8m27s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/vault-configurer 1/1 1 1 8m25s
deployment.apps/vault-operator 1/1 1 1 8m42s
NAME DESIRED CURRENT READY AGE
replicaset.apps/vault-configurer-7b8f4fd595 1 1 1 8m25s
replicaset.apps/vault-operator-59677fddb4 1 1 1 8m42s
NAME READY AGE
statefulset.apps/vault 1/1 8m25s
This is the behaviour i would expect actually since there is ownerReference relationship between the resources and the operator, all resources are eventually referenced to the Vault CRD
I did not have much time to look into this to be honest, but was planning to add some logging to the oeprator ( https://github.com/operator-framework/operator-sdk/blob/master/doc/user/logging.md ) to get some insight into deletion of resources
either the operator is deleting those resources ( and the logging could help ) or the ownership reference on your example is different from mine.
Could you double check that the ownership reference is as i described it on your side when it is all running as expected ?
For reference this is how they look like on my example (taken from octant )
VAULT-OPERATOR
VAULT-CRD
doggvalera
commented
4 years ago @primeroz Hi, adding to the @andreygolev - as he said, if the operator is manually scaled to 0 everything is ok, and Vault is working.
But if at that moment when the operator is scaled to 0 if all nodes lose the connection to Kubernetes API, all resources that were created by the vault-operator are gone.
primeroz
commented
4 years ago Oh ok, yeah i missed that detail :)
I ll test this when i get a chance, but if the resources are deleted even with the operator not running then it actually points to GC on kubernetes itself ! even scarier
primeroz
commented
4 years ago I was finally able to reproduce it on minikube.
Some kubelet logs :
apiserver is down, connection refused
Sep 26 08:09:31 minikube kubelet[3023]: E0926 08:09:31.989023 3023 reflector.go:125] object-"default"/"vault-tls": Failed to list *v1.Secret: Get https://localhost:8443/api/v1/namespaces/default/secrets?fieldSelector=metadata.name%3Dvault-tls&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused
Sep 26 08:09:32 minikube kubelet[3023]: E0926 08:09:32.388685 3023 reflector.go:125] object-"default"/"vault-configurer": Failed to list *v1.ConfigMap: Get https://localhost:8443/api/v1/namespaces/default/configmaps?fieldSelector=metadata.name%3Dvault-configurer&limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: connect: connection refused Apiserver is just back, the kubelet account get a couple of fobidden Did not see this when the kill of the apiserver was clean and did not trigger the recreation of resources. Related ? or just a byproduct of killing the apiserver since after 2 lines it stopped ?
Sep 26 08:09:35 minikube kubelet[3023]: E0926 08:09:35.964308 3023 reflector.go:125] object-"default"/"vault-configurer": Failed to list *v1.ConfigMap: configmaps "vault-configurer" is forbidden: User "system:node:minikube" cannot list resource "configmaps" in API group "" in the namespace "default": no relationship found between node "minikube" and this object
Sep 26 08:09:35 minikube kubelet[3023]: E0926 08:09:35.988439 3023 reflector.go:125] object-"default"/"vault-statsd-mapping": Failed to list *v1.ConfigMap: configmaps "vault-statsd-mapping" is forbidden: User "system:node:minikube" cannot list resource "configmaps" in API group "" in the namespace "default": no relationship found between node "minikube" and this objectRuntime errors are reported by kubelet 022e79d2b084301a4101d925ebb37764712312bc5ba3acdb9bb19ec5e6f72d34 is bank-vaults from configurer pod
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.231013 3023 remote_runtime.go:295] ContainerStatus "3858b3d9e5188cd4aa3af00b14037306900cceafd7903e1e49f9eb3696bb3d40" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 3858b3d9e5188cd4aa3af00b14037306900cceafd7903e1e49f9eb3696bb3d40
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.231965 3023 remote_runtime.go:295] ContainerStatus "91b8471bb30e340881238c9134e7ff4e50eea11716c8ec33aea4f83652e51276" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 91b8471bb30e340881238c9134e7ff4e50eea11716c8ec33aea4f83652e51276
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.232468 3023 remote_runtime.go:295] ContainerStatus "8c103572e87e904aeefc53f7c552626968a746694f6cdbcb635160b1d4d00c10" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 8c103572e87e904aeefc53f7c552626968a746694f6cdbcb635160b1d4d00c10
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.232961 3023 remote_runtime.go:295] ContainerStatus "939e2300974f1ca6d686b4e86636761c7cdc31fd52603892f498a21c860186bb" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 939e2300974f1ca6d686b4e86636761c7cdc31fd52603892f498a21c860186bb
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.246368 3023 remote_runtime.go:295] ContainerStatus "022e79d2b084301a4101d925ebb37764712312bc5ba3acdb9bb19ec5e6f72d34" from runtime service failed: rpc error: code = Unknown desc = Error: No such container: 022e79d2b084301a4101d925ebb37764712312bc5ba3acdb9bb19ec5e6f72d34
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.682259 3023 kubelet_pods.go:1093] Failed killing the pod "vault-configurer-85cc5c5c87-fzmnr": failed to "KillContainer" for "bank-vaults" with KillContainerError: "rpc error: code = Unknown desc = Error: No such container: 022e79d2b084301a4101d925ebb37764712312bc5ba3acdb9bb19ec5e6f72d34"
Process of tearing down pods start
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.265623 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-configurer" (UniqueName: "kubernetes.io/configmap/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-configurer") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62")
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.265668 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-tls" (UniqueName: "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-tls") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62")
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.265691 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-token-mkhsh" (UniqueName: "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-token-mkhsh") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62")
Sep 26 08:09:59 minikube kubelet[3023]: W0926 08:09:59.265926 3023 empty_dir.go:421] Warning: Failed to clear quota on /var/lib/kubelet/pods/57fc14d4-86aa-489c-9cce-7eb7696a9c62/volumes/kubernetes.io~configmap/vault-configurer: ClearQuota called, but quotas disabled
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.266103 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-configurer" (OuterVolumeSpecName: "vault-configurer") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62"). InnerVolumeSpecName "vault-configurer". PluginName "kubernetes.io/configma
p", VolumeGidValue ""
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.273060 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-token-mkhsh" (OuterVolumeSpecName: "vault-token-mkhsh") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62"). InnerVolumeSpecName "vault-token-mkhsh". PluginName "kubernetes.io/secret",
VolumeGidValue ""
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.276791 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-tls" (OuterVolumeSpecName: "vault-tls") pod "57fc14d4-86aa-489c-9cce-7eb7696a9c62" (UID: "57fc14d4-86aa-489c-9cce-7eb7696a9c62"). InnerVolumeSpecName "vault-tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.366157 3023 reconciler.go:297] Volume detached for volume "vault-configurer" (UniqueName: "kubernetes.io/configmap/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-configurer") on node "minikube" DevicePath ""
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.366236 3023 reconciler.go:297] Volume detached for volume "vault-tls" (UniqueName: "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-tls") on node "minikube" DevicePath ""
Sep 26 08:09:59 minikube kubelet[3023]: I0926 08:09:59.366254 3023 reconciler.go:297] Volume detached for volume "vault-token-mkhsh" (UniqueName: "kubernetes.io/secret/57fc14d4-86aa-489c-9cce-7eb7696a9c62-vault-token-mkhsh") on node "minikube" DevicePath ""
Sep 26 08:09:59 minikube kubelet[3023]: E0926 08:09:59.682259 3023 kubelet_pods.go:1093] Failed killing the pod "vault-configurer-85cc5c5c87-fzmnr": failed to "KillContainer" for "bank-vaults" with KillContainerError: "rpc error: code = Unknown desc = Error: No such container: 022e79d2b084301a4101d925ebb37764712312bc5ba3acdb9bb19ec5e6f72d34"
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.276354 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-file" (UniqueName: "kubernetes.io/host-path/baba9f0e-3dcf-415a-9d01-435d03c03bb7-pvc-a9b35465-ba43-459d-b343-59c26a0f06e7") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7")
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.276425 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-tls" (UniqueName: "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-tls") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7")
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.276477 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-token-mkhsh" (UniqueName: "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-token-mkhsh") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7")
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.276512 3023 reconciler.go:177] operationExecutor.UnmountVolume started for volume "vault-config" (UniqueName: "kubernetes.io/empty-dir/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-config") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7")
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.276839 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/baba9f0e-3dcf-415a-9d01-435d03c03bb7-pvc-a9b35465-ba43-459d-b343-59c26a0f06e7" (OuterVolumeSpecName: "vault-file") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7"). InnerVolumeSpecName "pvc-a9b35465-ba43-459d-b343-59c26a0
f06e7". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.286070 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-config" (OuterVolumeSpecName: "vault-config") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7"). InnerVolumeSpecName "vault-config". PluginName "kubernetes.io/empty-dir", VolumeGi
dValue ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.286364 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-tls" (OuterVolumeSpecName: "vault-tls") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7"). InnerVolumeSpecName "vault-tls". PluginName "kubernetes.io/secret", VolumeGidValue ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.286645 3023 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-token-mkhsh" (OuterVolumeSpecName: "vault-token-mkhsh") pod "baba9f0e-3dcf-415a-9d01-435d03c03bb7" (UID: "baba9f0e-3dcf-415a-9d01-435d03c03bb7"). InnerVolumeSpecName "vault-token-mkhsh". PluginName "kubernetes.io/secret",
VolumeGidValue ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.377903 3023 reconciler.go:297] Volume detached for volume "vault-tls" (UniqueName: "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-tls") on node "minikube" DevicePath ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.378915 3023 reconciler.go:297] Volume detached for volume "vault-token-mkhsh" (UniqueName: "kubernetes.io/secret/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-token-mkhsh") on node "minikube" DevicePath ""
Sep 26 08:10:01 minikube kubelet[3023]: I0926 08:10:01.379167 3023 reconciler.go:297] Volume detached for volume "vault-config" (UniqueName: "kubernetes.io/empty-dir/baba9f0e-3dcf-415a-9d01-435d03c03bb7-vault-config") on node "minikube" DevicePath ""
stale[bot]
commented
4 years ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
bonifaido
commented
4 years ago Guys, have you seen any update on this?
TomAdmz
commented
4 years ago Also seeing this issue on a few of our clusters running bank-vaults. We're seeing the same errors @andreygolev is seeing in our vault-operator. Just noticed this happening yesterday, and it occurred again last night - the pods don't show any restarts they are freshly created after this issue eventually resolves itself. Let me know what additional logs/info I can provide - Thanks!
andreygolev
commented
4 years ago In our case, I just took all resources created by vault-operator, put them to simple yamls and deployed them myself, then removed operator.
I don't remember if anyone told there about it, but this is Kubernetes garbabe collector removing resources created by vault operator, because there's no rerefence to them. Then vault operator recreates them back.
primeroz
commented
4 years ago @andreygolev just to understand , what reference are you talking about when you say that
Kubernetes garbabe collector removing resources created by vault operator, because there's no rerefence to them
In my tests the owner reference for all operator created objects was set to the Vault CRD, the CRD itself has no reference so is a top level object.
see pics in https://github.com/banzaicloud/bank-vaults/issues/649#issuecomment-531228165
In my tests that CRD never went away and its uuid never changed, so there would be no reason for Kubernetes GC to start a delete of all the child resources of it.
It definitely look like a kubernetes GC event though so i am curious to know if you are talking about that reference or something else and if maybe you saw something different from me ?
jengo
commented
4 years ago It appears that I might have been bit by this bug. After doing a kops upgrade, I noticed that my vault unseal keys are missing from k8s secrets. My backend is DynamoDB if that makes any difference.
Are there any config updates that can be used to never delete unseal keys ?
bonifaido
commented
4 years ago Hi @jengo, there is no such thing yet, but instead of using Kubernetes Secrets for storing the unseal keys I would suggest using S3 + KMS if you happen to be on AWS already.
sparqueur
commented
4 years ago I have the same issue. If I restart my cluster, vault is still present but after a few seconds it it killed and recreated. Unfortunately the secret is not here anymore :
Operator logs :
{"level":"info","ts":1581521554.836205,"logger":"cmd","msg":"Watched namespace: "}
{"level":"info","ts":1581521557.4336016,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8383"}
{"level":"info","ts":1581521557.4337976,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1581521559.9415529,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1581521559.9418552,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1581521577.441649,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"vault-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1581521577.6315415,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"vault-controller"}
{"level":"info","ts":1581521577.7369325,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"vault-controller","worker count":1}
{"level":"info","ts":1581521577.7384357,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}
{"level":"error","ts":1581521608.1343417,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"vault-controller","request":"test-vault/vault","error":"failed to distribute CA secret for vault: failed to create CA secret for vault in namespace test-keycloak: Timeout: request did not complete within requested timeout 30s","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1581521609.1445816,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}
{"level":"error","ts":1581521620.5421715,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"vault-controller","request":"test-vault/vault","error":"failed to distribute CA secret for vault: failed to query current secret for vault: Secret \"vault-tls\" not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1581521621.5427134,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}
{"level":"info","ts":1581521625.1354187,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}
{"level":"info","ts":1581521697.5353556,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}
{"level":"info","ts":1581521757.5356417,"logger":"controller_vault","msg":"Reconciling Vault","Request.Namespace":"test-vault","Request.Name":"vault"}Vault configurer logs :
time="2020-02-12T15:33:43Z" level=info msg="vault metrics exporter enabled: :9091/metrics"
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)
[GIN-debug] Listening and serving HTTP on :9091
time="2020-02-12T15:33:43Z" level=info msg="applying config file : /config/vault-configurer/vault-config.yml"
time="2020-02-12T15:33:43Z" level=info msg="checking if vault is sealed..."
time="2020-02-12T15:33:43Z" level=info msg="watching directory for changes: /config/vault-configurer/"
time="2020-02-12T15:33:47Z" level=error msg="error checking if vault is sealed: error checking status: Get https://vault.test-vault:8200/v1/sys/seal-status: dial tcp: lookup vault.test-vault on 10.43.0.10:53: no such host, waiting 5s before trying again..."
time="2020-02-12T15:33:52Z" level=info msg="checking if vault is sealed..."
time="2020-02-12T15:35:26Z" level=error msg="error checking if vault is sealed: error checking status: Get https://vault.test-vault:8200/v1/sys/seal-status: dial tcp 10.43.165.196:8200: i/o timeout, waiting 5s before trying again..."
time="2020-02-12T15:35:31Z" level=info msg="checking if vault is sealed..."
time="2020-02-12T15:37:05Z" level=error msg="error checking if vault is sealed: error checking status: Get https://vault.test-vault:8200/v1/sys/seal-status: dial tcp 10.43.165.196:8200: i/o timeout, waiting 5s before trying again..."Little complement to my previous message. The problem is that :
3 simple solutions for now :
I think tha Opt1 is going to take a long time to determine why the operator determines it needs to destroy / recreate the Vault ==> Would it be possible to temporally remove the ownerReference (Opt2)
Do you think it might be possible to generate a new secret if it does not exist instead of checking if it is the first time it runs ? Not sure about the ability of having a new unseal secret value
Thanks in advance
sparqueur
commented
4 years ago With debug logs : Seems to be an issue with the tls secret
2020-02-12T18:17:53.952Z INFO cmd Watched namespace:
2020-02-12T18:17:56.505Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8383"}
2020-02-12T18:17:56.505Z INFO cmd Registering Components.
2020-02-12T18:17:59.011Z INFO cmd Starting the Cmd.
2020-02-12T18:17:59.045Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-02-12T18:18:16.652Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"test-vault","name":"vault-operator-lock","uid":"fb66b9ff-ce9d-499a-b445-c4249086f26a","apiVersion":"v1","resourceVersion":"1459473"}, "reason": "LeaderElection", "message": "test-vault-vault-operator-8697cd5f68-fnqsl_b2b80712-c575-4311-b24f-9966e982e9df became leader"}
2020-02-12T18:18:16.652Z INFO controller-runtime.controller Starting EventSource {"controller": "vault-controller", "source": "kind source: /, Kind="}
2020-02-12T18:18:16.844Z INFO controller-runtime.controller Starting Controller {"controller": "vault-controller"}
2020-02-12T18:18:16.949Z INFO controller-runtime.controller Starting workers {"controller": "vault-controller", "worker count": 1}
2020-02-12T18:18:16.951Z INFO controller_vault Reconciling Vault {"Request.Namespace": "test-vault", "Request.Name": "vault"}
2020-02-12T18:18:17.055Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:,APIVersion:,}:vault
2020-02-12T18:18:17.055Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Service,APIVersion:v1,}:vault-0
2020-02-12T18:18:17.252Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls
2020-02-12T18:18:17.357Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"test-vault\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}]},\\\"stringData\\\":{\\\"ca.crt\\\":\\\"-----BEGIN CERTIFICATE-----\\\\nMIIDTTCCAjWgAwIBAgIRAPQ8czzvNKgCN1HKUd6aW4IwDQYJKoZIhvcNAQELBQAw\\\\nQDEVMBMGA1UEChMMQmFuemFpIENsb3VkMScwJQYDVQQDEx5CYW56YWkgQ2xvdWQg\\\\nR2VuZXJhdGVkIFJvb3QgQ0EwHhcNMjAwMjEyMTczOTE4WhcNMjEwMjExMTczOTE4\\\\nWjBAMRUwEwYDVQQKEwxCYW56YWkgQ2xvdWQxJzAlBgNVBAMTHkJhbnphaSBDbG91\\\\nZCBHZW5lcmF0ZWQgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\\\\nggEBALIFpqus2/xyUvdGfn6u4qje4tdHX0JSl1ybE7AjY8P2LoeXwHT5UaVtY38j\\\\nC9Umztg51Bbd7NedPJE14kxHpxHBCfcoJOUrP5+pcvEjAGBwNJyOvC+ZwzYPp457\\\\nnBSSWa7ikO814NbnJPGcBbLKp4U8DrxNJIRY7t7F2h4L3ybLMorJwew4/G2XmI2F\\\\ndlYeh2lMtxVyg4pnEPGt8P3lMHJZ0m5nLquNr5gZzRoEo6Q8KbR0wk/QhnBFyoj0\\\\nDY9eShL7zEkca7pL4x7nJXrVciW7W16xZg2+gJFg4s/EsHumR8ACsInqvCiILW5t\\\\nJYjoAj5Hwi8BqC3LZtoVmNmalz0CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgIEMB0G\\\\nA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/MA0G\\\\nCSqGSIb3DQEBCwUAA4IBAQArfyNTP7LsJhY3A4AVLBv/zkwJu9fbIvmhtIRmzR/H\\\\n9olW3PqBQGU+yGJWdH7fFXKe+XcK3+cJhpBxxLc16cIrWRZJpkLx20TMmOpU3xPJ\\\\n2cCdXgcDsp7bHYWCKEn/PBBvwD/UmFNT+kkz/AVwp6caWZh+VRH2trK3cFVnvQs/\\\\noAbQ2Os+d1qLX9d3mvwwpbL3L1vkTW7QNGq
dzku/1h/OAd77pua/eSskxvTSnpS6\\\\nehbWiXnr0oc1ImyfO9ijCFeYFjuj/SQMH3v+rF7kW/toAXgEHZo6Y6McG2Dy67Gl\\\\nQcDDMImBTPh1/LIeNYrc+cFMmEr7iNrg0h5Yq+Vu5fe1\\\\n-----END CERTIFICATE-----\\\\n\\\",\\\"server.crt\\\":\\\"-----BEGIN CERTIFICATE-----\\\\nMIIDlDCCAnygAwIBAgIRAN+92LRxcmZE9blfZX6//+UwDQYJKoZIhvcNAQELBQAw\\\\nQDEVMBMGA1UEChMMQmFuemFpIENsb3VkMScwJQYDVQQDEx5CYW56YWkgQ2xvdWQg\\\\nR2VuZXJhdGVkIFJvb3QgQ0EwHhcNMjAwMjEyMTczOTE4WhcNMjEwMjExMTczOTE4\\\\nWjBEMRUwEwYDVQQKEwxCYW56YWkgQ2xvdWQxKzApBgNVBAMTIkJhbnphaSBDbG91\\\\nZCBHZW5lcmF0ZWQgU2VydmVyIENlcnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\\\\nggEKAoIBAQCujdZJvvwPHhbs29XDGBLczdH6aGF14GiiMgzjWcAyYsqm/ZIYqpdb\\\\nOFDVFcSzRSrHqcql0S5BfJ6P+YucckEvDTe/jrIwyyKz/B90Yj3ePVlsBT3hMHH6\\\\n/NBihiJ5SxhjzW1SDDzFNpfx3mmhj/iv2AlDtrB1roPpHz5VTiV/MoQMkapDlWJc\\\\nJbI5Yf8GkAa4sYzaKTR26utM/yBKTRDBP46c8OIZANNTJcu/Gl2wHRKiNBiTUVVB\\\\nSaMH+aGAHKi3e0Yq/9AUeH8ThB9SuDMSNqbTJCzmCLoQj7Wu3VZyc2DofCe4CH4a\\\\n1/rpCBsMfbipSE9UKkGYqI6hs3d2PjLRAgMBAAGjgYQwgYEwDgYDVR0PAQH/BAQD\\\\nAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwTAYDVR0RBEUw\\\\nQ4IFdmF1bHSCEHZhdWx0LnRlc3QtdmF1bHSCInZhdWx0LnRlc3QtdmF1bHQuc3Zj\\\\nLmNsdXN0ZXIubG9jYWyHBH8AAAEwDQYJKoZIhvcNAQELBQADggEBAGcjQOrn8BDn\\\\nCtsWrHQShOMM7fqmZXQFLQLEhp6zo1iTi/MS03wYcwZQb85Nptf5pQCGM2VdpWxR\\\\nzbg3bQ4vTQG26N1ydpKLgoERorf4KSmlNSRxYFHlfIvF+HqE6c0yFl7l4fkhjZni\\\\nm+NRQTdq6uf9PCoNfxGufcCjZC1wFT73V+cmQiyGNX05K6xfu45BkqBKgrQ58bGN\\\\neITfABMv4P7hTKJn5+aWzKwAcqjW8KR3jXzcNjJPDdk/F/0VZhZLliXvNK2Y69wK\\\\n9ukzME/NG3oYLG5RhcQkthgxJcycxy0j7/QVrI5+8kjDdoworz0/fCc6sAa7dzRd\\\\nit+2DGtZLto=\\\\n-----END CERTIFICATE-----\\\\n\\\",\\\"server.key\\\":\\\"-----BEGIN RSA PRIVATE KEY-----\\\\nMIIEpAIBAAKCAQEAro3WSb78Dx4W7NvVwxgS3M3R+mhhdeBoojIM41nAMmLKpv2S\\\\nGKqXWzhQ1RXEs0Uqx6nKpdEuQXyej/mLnHJBLw03v46yMMsis/wfdGI93j1ZbAU9\\\\n4TBx+vzQYoYieUsYY81tUgw8xTaX8d5poY/4r9gJQ7awda6D6R8+VU4lfzKEDJGq\\\\nQ5ViXCWyOWH/BpAGuLGM2ik0durrTP8gSk0QwT+OnPDiGQDTUyXLvxpdsB0SojQY\\\\nk1FVQUmjB/mhgByot3tGKv/QFHh/E4QfUrgzEjam0yQs5gi6EI+1rt1WcnNg6Hwn\\\\nuAh+Gtf66QgbDH24qUhPVCpBmKiOobN3dj4y0QI
DAQABAoIBAQCYtrzo7i9mJoQW\\\\nG8rsEc7G4Uc5FkNWNsw0QyelbKtLKcV0zdoYvH8JLr8BfjcGRJhF8NW7tsQKRTYQ\\\\nblnnuMowmRbR/80EBWTKJOV8FEteeTwX1oopzxGZbb/+wq28gqfHAT4Ij25HeBj9\\\\nYgH2eJgxeAe690Vw4ArGVTBav7Dsk1qdTsE2I2qjxHjKu1W8Xr5jnws/tHlLoC2J\\\\nS/PAkX+5t1wgQp4nLoqhm8kZrkNdYWEUESovJfKNHIikM9dzsDhgJoco+Qtwnnsv\\\\nbfAnm28x9yaCXzRJ1ZngarJuAOBOqo8uccy5KIafCn0oKruEYHj4tspnLvkCM5BD\\\\nEc0fVfpFAoGBANfwNu0DrqpSJxbmRHY6RNN9efvY0EFsGnFnY+JyrYNtZ0WAYcFo\\\\nPsqQZQgSijf/G++6HVKnIArdxI2KZ+SjNk+HE81z/6U45d+1LNd2l0fe3PAaaPX/\\\\neMbqGQV8d39INiFgMUfdvbbQnX9+d97WG4+c5RR83HwBhHfZF+OiSitXAoGBAM7w\\\\nHYGZloSkv+MC1k8uQkjxMVRjKy4u1aHfdMbevTOh8FfCBbvssfh47fsiVbKriB11\\\\nH3lWA03d2M8/GqmJZ4reGrlj4MOwu6jf4guHVZiATyb97bRLlGAwtTXBMxinGCjV\\\\nSwQ8nrWX4n9MOI5nryQy4UalaqP8v8DVMsF5y2IXAoGANjAjuGqaJEelY4u5Vmnc\\\\n562vNNH9wM0d/aNSX3sjq72CkuBap6w/RZnioJn2cYLlQc1Umm8yDUgFkPdQhwGm\\\\nGWLuRzp6xr12giL288Lgx7pXezFFcaYAavpH+SZ43koCu3FmLICqoPvM6jeUYSkd\\\\nlXZrzi5fLgc7dM/XvNn2/tUCgYBPLEr9tLeD1EyeHA3GiZpOsgj/klmI995AanAR\\\\nTlLxGDoOOWtcNPj6aboTZVWIYFxZeOAr2h61gfVbhXc7+hglZ0iuOWpbNJj3yYCA\\\\njKzRBfZCotB5NJl2G+yYvmxAOF3j8Ycb/d1y+MNtlMnmogC+RsEt+EfOqfrrWiZr\\\\nI6AOzQKBgQDBakfW9bsUAt2Mt68SH06Jawn4jY8fL1AwCKjLPqJFs4qeysT5tNJu\\\\nrp5onb1zz3D2xeC2SfAq+PfjMHmAkDSYjjoc2Qagg+Kyr0Fep8k9x9ceCvjrx2EP\\\\nf6+dR75Z3ik9IWjoHDHAJ+g8uSlH87EajH1isIfN59Lre6AFVhq83w==\\\\n-----END RSA PRIVATE KEY-----\\\\n\\\"}}\"},\"creationTimestamp\":\"2020-02-12T17:39:29Z\",\"selfLink\":\"/api/v1/namespaces/test-vault/secrets/vault-tls\"}}"}
2020-02-12T18:18:47.359Z ERROR controller-runtime.controller Reconciler error {"controller": "vault-controller", "request": "test-vault/vault", "error": "failed to distribute CA secret for vault: failed to create CA secret for vault in namespace cattle-system: Timeout: request did not complete within requested timeout 30s"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-02-12T18:18:48.444Z INFO controller_vault Reconciling Vault {"Request.Namespace": "test-vault", "Request.Name": "vault"}
2020-02-12T18:19:00.945Z ERROR controller-runtime.controller Reconciler error {"controller": "vault-controller", "request": "test-vault/vault", "error": "failed to distribute CA secret for vault: failed to query current secret for vault: Secret \"vault-tls\" not found"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-02-12T18:19:02.044Z INFO controller_vault Reconciling Vault {"Request.Namespace": "test-vault", "Request.Name": "vault"}
2020-02-12T18:19:02.045Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:,APIVersion:,}:vault
2020-02-12T18:19:02.045Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Service,APIVersion:v1,}:vault-0
2020-02-12T18:19:02.047Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls
2020-02-12T18:19:05.368Z DEBUG controller_vault Updating vault status {"status": {"nodes":["vault-0"],"leader":""}, "resourceVersion": "1454433"}
2020-02-12T18:19:05.447Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "vault-controller", "request": "test-vault/vault"}
2020-02-12T18:19:05.544Z INFO controller_vault Reconciling Vault {"Request.Namespace": "test-vault", "Request.Name": "vault"}
2020-02-12T18:19:05.645Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:,APIVersion:,}:vault
2020-02-12T18:19:05.646Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Service,APIVersion:v1,}:vault-0
2020-02-12T18:19:05.647Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls
2020-02-12T18:19:05.750Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"test-vault\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}]},\\\"stringData\\\":{\\\"ca.crt\\\":\\\"-----BEGIN CERTIFICATE-----\\\\nMIIDTTCCAjWgAwIBAgIRAOh/hIaPqraKfqKSH6+qa34wDQYJKoZIhvcNAQELBQAw\\\\nQDEVMBMGA1UEChMMQmFuemFpIENsb3VkMScwJQYDVQQDEx5CYW56YWkgQ2xvdWQg\\\\nR2VuZXJhdGVkIFJvb3QgQ0EwHhcNMjAwMjEyMTgxODQ4WhcNMjEwMjExMTgxODQ4\\\\nWjBAMRUwEwYDVQQKEwxCYW56YWkgQ2xvdWQxJzAlBgNVBAMTHkJhbnphaSBDbG91\\\\nZCBHZW5lcmF0ZWQgUm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC\\\\nggEBANahdahI1qbi8oZSOA0AlfvBjr+xOTne8wiNb6PL7hAMAX+lxYpmSrV7qJI9\\\\nZya/prELx/l/kw7Ch8PBlGdQUM53hZ2GANQHtJtSPrtMWN5LqNHP83l9ncKwe8Vs\\\\ndtv6g06cTWGwrlJBxqJakcdF+fwH2uUeR9uYYZACokCavB9iU2GpzoHOOOEpNR2c\\\\nmWueaWH7yDmqZBP1hS7foJuxDsZwZVRpa5jsdylrf7GrEaEQP1hcY5CORPqSVFRs\\\\nkVBuisjtirj4N3d7i3lJh6se9iLzMt1rCvN0Nzaj21tX2gpg1jDnq4SUZxNugDjp\\\\nuRoDr3fdr9zW1BR0KlYXbuMv9d0CAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgIEMB0G\\\\nA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/MA0G\\\\nCSqGSIb3DQEBCwUAA4IBAQBG7HeJcsgbRKWXMo+CHaD0wdG0F6G5hdQ+ISNf5jIf\\\\nw8QlcXWPG6fwudX6piYBssOWMm584OM/S+yPY79wuzvj3GMmZCWK/UAEVA818vXB\\\\nndUDKRJQgeDYdMucUgWvJzyQHU09qI55X6Ro+M7+zkhe5VBbyGGt+HQcWBhC+x3I\\\\nkOlFySsXjGG+ibvBbiHFBOkTTzuyx81dwRj
KeGAmYSGVq7vk09ton3hRZcou4Ba5\\\\nIKrK98/GOxarXY+pBivZl2i8s8Aki+BUu84ZS90ksTzxohSRRnW2iMnngQlrYG22\\\\nC8NxnzGynxS2jFHNRP7e6mkzsswBccT/lcBefuAhxC/x\\\\n-----END CERTIFICATE-----\\\\n\\\",\\\"server.crt\\\":\\\"-----BEGIN CERTIFICATE-----\\\\nMIIDlDCCAnygAwIBAgIRAMViu9HlIxyklpJwVWCQSjswDQYJKoZIhvcNAQELBQAw\\\\nQDEVMBMGA1UEChMMQmFuemFpIENsb3VkMScwJQYDVQQDEx5CYW56YWkgQ2xvdWQg\\\\nR2VuZXJhdGVkIFJvb3QgQ0EwHhcNMjAwMjEyMTgxODQ4WhcNMjEwMjExMTgxODQ4\\\\nWjBEMRUwEwYDVQQKEwxCYW56YWkgQ2xvdWQxKzApBgNVBAMTIkJhbnphaSBDbG91\\\\nZCBHZW5lcmF0ZWQgU2VydmVyIENlcnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw\\\\nggEKAoIBAQC9Z6/E1GSKr0xcF6roLJLWVrQfQOCUnWBeM2tdEse/f2wGtgIeiDAC\\\\nc2IeUZxyj2Scmov9zc7VZADT1BwiT5CuiVn2VCLP5Hjvbncm1yHLby4jYc1tFLCm\\\\nW8caKfXL0pRBgkShIZJ2b3lfa8Zd/w/Jl3YnrBfSM1vpjllWn0dz1YYHfPx/qKFW\\\\nrZ9KQw30FlktYBDd8VV8hxjpRImpb1X3Y6T8/LTEiIictFzTqqN6ybuwW16JaYu+\\\\n/wtGfnyDB4qFQ6xuiLA0umMdwEXI6fAffR5eE0rvo3npzVHZ0+W6MrpGLurEnm7J\\\\nkC1ESMO6bDXF5ZtmERTWGqib9pFNLKAzAgMBAAGjgYQwgYEwDgYDVR0PAQH/BAQD\\\\nAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwTAYDVR0RBEUw\\\\nQ4IFdmF1bHSCEHZhdWx0LnRlc3QtdmF1bHSCInZhdWx0LnRlc3QtdmF1bHQuc3Zj\\\\nLmNsdXN0ZXIubG9jYWyHBH8AAAEwDQYJKoZIhvcNAQELBQADggEBAAJn2NeqI57B\\\\nPHy3xbC3PURR6uuQUiJM7kTw5ji55DJMLxQ9G7ZoV7mEjRS9gDwxP4tvYu79ZQOh\\\\nPDMtKbNpg2hrzBQ8slLa3iGhE6R+QHdg0r2OAO/08jaB1vI1FxkY5aaEQXBRlUx/\\\\n4J4wpCK0esySn2wZGhevlTIFhpIonT3ivmqzPrcmzo/9SkQ+FZacY6ZKor6ITYe0\\\\nTYiyFezOaGxHT1CMRhZgmlHvHZnVM/XW5pBGTSOi4RaXxH7u8PE1RMjFAdOBd3gi\\\\nqLarfxGRBSBO3X2WYU25V1benONXv8zhpNDIXaX2UO6Z2oHcG9pZJtzV5rf2uma3\\\\nDHZ/f5bJ6JE=\\\\n-----END CERTIFICATE-----\\\\n\\\",\\\"server.key\\\":\\\"-----BEGIN RSA PRIVATE KEY-----\\\\nMIIEpAIBAAKCAQEAvWevxNRkiq9MXBeq6CyS1la0H0DglJ1gXjNrXRLHv39sBrYC\\\\nHogwAnNiHlGcco9knJqL/c3O1WQA09QcIk+QrolZ9lQiz+R47253Jtchy28uI2HN\\\\nbRSwplvHGin1y9KUQYJEoSGSdm95X2vGXf8PyZd2J6wX0jNb6Y5ZVp9Hc9WGB3z8\\\\nf6ihVq2fSkMN9BZZLWAQ3fFVfIcY6USJqW9V92Ok/Py0xIiInLRc06qjesm7sFte\\\\niWmLvv8LRn58gweKhUOsboiwNLpjHcBFyOnwH30eXhNK76N56c1R2dPlujK6Ri7q\\\\nxJ5uyZAtREjDumw1xeWbZhEU1hqom/aRTSygMwI
DAQABAoIBAH5Ik7j4o9CT8KW2\\\\nYg1LKtweExqDdzmfdJFaV0tHKkdT/gK7d85Dnz8DD3iHkXgQg1Sm9o/SmQVOF/8k\\\\nfV2jDyRQD9tdXZifDMMXM/qBlda92lVc8Eq6cM16g9W+BFRRoufKo1eLusucp6An\\\\ni6tAVJZkljL9WySbebzPXGGU8V8Y6PkPm3OT0ixL4oviRMpgqXyuo1qh3Tydj6mZ\\\\nfPsE81AQjTbHgMNo6gf1UUlB6CgYorApnzM0rDN8GaISxHuL7zdBNeblZilNtF3y\\\\nS2VlXMlI4ZRGf3+M0pzDXQvPQjPUAajlRPTs3IwQnmf8LdbjgeaYjT9JvEE9xbfd\\\\nZLSKsYECgYEA1qHbLbRiRascAMNEi/9F+5eyeCx7FO6QhCb2DgN5KIwspfZ3lySu\\\\nRIcfEdmy55mimwDZ+wph6uGD0gWTX6cWQnWpUOVYjuuUR4Ly5IG8Lu/VuKeLYpyJ\\\\nYDBcWHVLNoM7jGxJP0AxxRJFamXd6p5WzUabkSiyeIFv+dkEbv9dKn0CgYEA4ekY\\\\n1E5TWqNsyWoAMORgs0b73jFibV7X201caVfFuW9REkrr5xkv9T68+96TzOGd7Mvu\\\\nCQ6dVROdFITOY+xAJXPmlWs4HXM2/pI7p9FwS9VZt9g6vRNhnD771lbep4KTgLv0\\\\nNkjLzIjQdvviDKXfFQK+QNfnTr6gVKVOZJDWRG8CgYEAgV/vDg29buFFb/BEUV7+\\\\ngDlgiq/FQUjMM9yIVMojLh3Qi/mU/LZPlV8di1T+Ujx5l5Vbz8biXwKwwtg5jV7a\\\\nPBVQikK0rvGx36k/YB4ToPdQjI3xpu9Un3aHM4sAW0ufiXV8cXPwjCoHTas5qULo\\\\nb7Qv8b/mVF0xchkuYvnq3sECgYBMNvW3eo1IgOYMtYRG+mVdVk9W0NZiC6NLSzgo\\\\nGgKM4u84+qnHl4nEdVwQpTC76En9P3eY+qM9tpZB++QddfDWMdnTAw81cE9XZ0JV\\\\nBaWbD+lqyrpWW5mMuvr/fuMzJCxbPA3R2pFW+blKywOiPLucPQaunahdupYguHxQ\\\\nFUUUZQKBgQDGDLqYiPtZsoO1zIYKEt1ubM45W/vPa0vhQqWqyrQynlmsOPYnDflQ\\\\nTLlnpkJIEKXw5Nz435qqV10WZKlw/uTdlG/TN+/hlmxXWr8zsE4uhyTXmyZ+u8ir\\\\nt/8+OUvQRWKfrANLamIL8EqoSFLviwT2KhXR6umNg4fEobJWijt9xQ==\\\\n-----END RSA PRIVATE KEY-----\\\\n\\\"}}\"},\"creationTimestamp\":\"2020-02-12T18:19:00Z\",\"selfLink\":\"/api/v1/namespaces/test-vault/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.759Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:00Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/test-vault/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/default/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.771Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"kube-public\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/default/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/kube-public/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.848Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"test-nexus\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/kube-public/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/test-nexus/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.859Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"ingress-nginx\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/test-nexus/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/ingress-nginx/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.869Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"kube-system\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/ingress-nginx/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/kube-system/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.945Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"test-registrytrustmanager\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/kube-system/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/test-registrytrustmanager/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.958Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"cattle-system\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/test-registrytrustmanager/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/cattle-system/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.968Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\"
true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"kube-node-lease\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/cattle-system/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/kube-node-lease/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.978Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Secret,APIVersion:v1,}:vault-tls {"patch": "{\"apiVersion\":\"v1\",\"kind\":\"Secret\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"data\\\":{\\\"ca.crt\\\":\\\"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUVENDQWpXZ0F3SUJBZ0lSQU9oL2hJYVBxcmFLZnFLU0g2K3FhMzR3RFFZSktvWklodmNOQVFFTEJRQXcKUURFVk1CTUdBMVVFQ2hNTVFtRnVlbUZwSUVOc2IzVmtNU2N3SlFZRFZRUURFeDVDWVc1NllXa2dRMnh2ZFdRZwpSMlZ1WlhKaGRHVmtJRkp2YjNRZ1EwRXdIaGNOTWpBd01qRXlNVGd4T0RRNFdoY05NakV3TWpFeE1UZ3hPRFE0CldqQkFNUlV3RXdZRFZRUUtFd3hDWVc1NllXa2dRMnh2ZFdReEp6QWxCZ05WQkFNVEhrSmhibnBoYVNCRGJHOTEKWkNCSFpXNWxjbUYwWldRZ1VtOXZkQ0JEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQwpnZ0VCQU5haGRhaEkxcWJpOG9aU09BMEFsZnZCanIreE9UbmU4d2lOYjZQTDdoQU1BWCtseFlwbVNyVjdxSkk5Clp5YS9wckVMeC9sL2t3N0NoOFBCbEdkUVVNNTNoWjJHQU5RSHRKdFNQcnRNV041THFOSFA4M2w5bmNLd2U4VnMKZHR2NmcwNmNUV0d3cmxKQnhxSmFrY2RGK2Z3SDJ1VWVSOXVZWVpBQ29rQ2F2QjlpVTJHcHpvSE9PT0VwTlIyYwptV3VlYVdIN3lEbXFaQlAxaFM3Zm9KdXhEc1p3WlZScGE1anNkeWxyZjdHckVhRVFQMWhjWTVDT1JQcVNWRlJzCmtWQnVpc2p0aXJqNE4zZDdpM2xKaDZzZTlpTHpNdDFyQ3ZOME56YWoyMXRYMmdwZzFqRG5xNFNVWnhOdWdEanAKdVJvRHIzZmRyOXpXMUJSMEtsWVhidU12OWQwQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0lFTUIwRwpBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFCRzdIZUpjc2diUktXWE1vK0NIYUQwd2RHMEY2RzVoZFErSVNOZjVqSWYKdzhRbGNYV1BHNmZ3dWRYNnBpWUJzc09XTW01ODRPTS9TK3lQWTc5d3V6dmozR01tWkNXSy9VQUVWQTgxOHZYQgpuZFVES1JKUWdlRFlkTXVjVWdXdkp6eVFIVTA5cUk1NVg2Um8rTTcremtoZTVWQmJ5R0d0K0hRY1dCaEMreDNJCmtPbEZ5U3NYakdHK2lidkJiaUhGQk9rVFR6dXl4ODFkd1JqS2VHQW1ZU0dWcTd2azA5dG9uM2hSWmNvdTRCYTUKSUtySzk4L0dPeGFyWFkrcEJpdlpsMmk4czhBa2krQlV1ODRaUzkwa3NUenhvaFNSUm5XMmlNbm5nUWxyWUcyMgpDOE54bnpHeW54UzJqRkhOUlA3ZTZta3pzc3dCY2NUL2xjQmVmdUFoeEMveAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==\\\"},\\\"kind\\\":\\\"Secret\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"common/annotation\\\":\\\
"true\\\",\\\"type/instance\\\":\\\"vault\\\"},\\\"creationTimestamp\\\":\\\"2020-02-12T18:19:02Z\\\",\\\"labels\\\":{\\\"app.kubernetes.io/name\\\":\\\"vault\\\",\\\"example.com/log-format\\\":\\\"json\\\",\\\"vault_cr\\\":\\\"vault\\\"},\\\"name\\\":\\\"vault-tls\\\",\\\"namespace\\\":\\\"test-catrustmanager\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"vault.banzaicloud.com/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"Vault\\\",\\\"name\\\":\\\"vault\\\",\\\"uid\\\":\\\"0143226e-6b1c-4709-87d4-823f1edf714e\\\"}],\\\"selfLink\\\":\\\"/api/v1/namespaces/kube-node-lease/secrets/vault-tls\\\"},\\\"type\\\":\\\"Opaque\\\"}\"},\"selfLink\":\"/api/v1/namespaces/test-catrustmanager/secrets/vault-tls\"}}"}
2020-02-12T18:19:05.985Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:ConfigMap,APIVersion:v1,}:vault-statsd-mapping
2020-02-12T18:19:06.044Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:StatefulSet,APIVersion:apps/v1,}:vault
2020-02-12T18:19:06.045Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:ConfigMap,APIVersion:v1,}:vault-configurer
2020-02-12T18:19:06.046Z DEBUG controller_vault Resource update for object &TypeMeta{Kind:Deployment,APIVersion:apps/v1,}:vault-configurer {"patch": "{\"spec\":{\"template\":{\"spec\":{\"$setElementOrder/containers\":[{\"name\":\"bank-vaults\"}],\"$setElementOrder/volumes\":[{\"name\":\"vault-configurer\"},{\"name\":\"vault-tls\"}],\"containers\":[{\"$setElementOrder/volumeMounts\":[{\"mountPath\":\"/config/vault-configurer\"},{\"mountPath\":\"/vault/tls\"}],\"args\":[\"--mode\",\"k8s\",\"--k8s-secret-namespace\",\"test-vault\",\"--k8s-secret-name\",\"vault-unseal-keys\",\"--vault-config-file\",\"/config/vault-configurer/vault-config.yml\"],\"name\":\"bank-vaults\",\"volumeMounts\":[{\"mountPath\":\"/config/vault-configurer\",\"name\":\"vault-configurer\"}]}],\"volumes\":[{\"configMap\":{\"name\":\"vault-configurer\"},\"name\":\"vault-configurer\"}]}}}}"}
2020-02-12T18:19:06.055Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Service,APIVersion:v1,}:vault-configurer
2020-02-12T18:19:06.055Z DEBUG controller_vault Skipping update for object &TypeMeta{Kind:Ingress,APIVersion:extensions/v1beta1,}:vault
2020-02-12T18:19:08.070Z DEBUG controller-runtime.controller Successfully Reconciled {"controller": "vault-controller", "request": "test-vault/vault"}Hi there, I still think that this is a Kubernetes bug since the root ownerReference of the Secret (and all components created by the operator) is the Vault CR itself and not the Pod. You can easily check if you get the details of the Secret created:
$ kubectl get secret -o yaml vault-unseal-keys
apiVersion: v1
kind: Secret
metadata:
creationTimestamp: "2020-02-13T10:07:04Z"
name: vault-unseal-keys
namespace: default
ownerReferences:
- apiVersion: vault.banzaicloud.com/v1alpha1
controller: true
kind: Vault
name: vault
uid: 7bbea4d6-4e48-11ea-871b-0a14a3667e10
resourceVersion: "204297"
selfLink: /api/v1/namespaces/default/secrets/vault-unseal-keys
uid: 958c4441-4e48-11ea-9eaf-0608e1b93744
type: Opaque
data:
...Anyhow the ownerReference setting has been removed from the code a few days ago for various other data protection reasons: https://github.com/banzaicloud/bank-vaults/pull/866 and it is in master already (not yet in a released version). This should resolve this issue I think.
sparqueur
commented
4 years ago Hi @bonifaido, You are right about the ownerReferences, sercret should not have been deleted, my bad. Thanks for the fix, Any idea when the next release will be scheduled ?
bonifaido
commented
4 years ago No problem!
I can cut an RC for this anytime, but the next release will be around roughly 2 weeks from now on, in the beginning of March.
Describe the bug: Vault resources are recreated after Vault operator temporarily loses connection to Kubernetes API. For example:
We see errors in vault-operator pod that tell that there's no master.
Then we fix Kubernetes API master, and it's up and running again. We don't see anymore error messages in vault-operator.
Everything looks fine. During one minute we notice that our Vault resources are fully recreated, including statefulset, pods, vault-tls secret. Like, all resources that were created by vault-operator. CR itself is untouched. Timestamp of CR is the same as on date of creation.
Expected behaviour: Vault resources are not being recreated after temporary loss of connection with Kubernetes Master
Steps to reproduce the bug: It's easy to reproduce bug on minikube.
minikube sshsudo pkil kube-apiserverIt's not always happens at first try. Sometimes I had to kill API server one more time if I don't see that vault resources are recreated.
Note: I had to adjust operator-rbac.yaml because it contains not all required permissions for operator.
And had to create rbac for vault itself:
Additional context: Just more logs that appear after we kill Kubernetes API server
Environment details:
First it happened in Azure AKS, but reproducible on Minikube. So, seems to be that it applies to all environments.
Thank you!
/kind bug