tibordp
commented
3 years ago Hey @samip5, the manifest alone ought to be enough, kube-webhook-certgen jobs should automatically generate any secrets needed.
Did it not work put of the box?
Open samip5 opened 3 years ago
tibordp
commented
3 years ago Hey @samip5, the manifest alone ought to be enough, kube-webhook-certgen jobs should automatically generate any secrets needed.
Did it not work put of the box?
samip5
commented
3 years ago Hey @samip5, the manifest alone ought to be enough, kube-cert-gen jobs should automatically generate any secrets needed.
Did it not work put of the box?
It didin't. I'm using cert-manager.
core False apply failed: Error from server (Invalid): error when applying patch: main/5eed6a4d4eb4c7778e104c1e72a8b90ad12b81bc False
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"kustomize.toolkit.fluxcd.io/name\":\"core\",\"kustomize.toolkit.fluxcd.io/namespace\":\"flux-system\"},\"name\":\"prefer-dual-stack-admission-create\",\"namespace\":\"prefer-dual-stack\"},\"spec\":{\"template\":{\"metadata\":{\"name\":\"prefer-dual-stack-admission-create\"},\"spec\":{\"containers\":[{\"args\":[\"create\",\"--host=prefer-dual-stack-webhook,prefer-dual-stack-webhook.$(POD_NAMESPACE).svc\",\"--namespace=$(POD_NAMESPACE)\",\"--secret-name=prefer-dual-stack-admission\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"docker.io/jettech/kube-webhook-certgen:v1.5.2\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"create\"}],\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"prefer-dual-stack-admission\"}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"create"}],"containers":[{"image":"docker.io/jettech/kube-webhook-certgen:v1.5.2","name":"create"}]}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "prefer-dual-stack-admission-create", Namespace: "prefer-dual-stack"
for: "08947f31-e8f4-40a9-a646-8de1de9b36bf.yaml": Job.batch "prefer-dual-stack-admission-create" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"prefer-dual-stack-admission-create", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"84f691ea-0955-4fb4-b645-1ffcefedd185", "job-name":"prefer-dual-stack-admission-create"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create", Image:"docker.io/jettech/kube-webhook-certgen:v1.5.2", Command:[]string(nil), Args:[]string{"create", "--host=prefer-dual-stack-webhook,prefer-dual-stack-webhook.$(POD_NAMESPACE).svc", "--namespace=$(POD_NAMESPACE)", "--secret-name=prefer-dual-stack-admission"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0x401fd78bc0)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0x400fd5a9e8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"prefer-dual-stack-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0x402005ca00), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"kustomize.toolkit.fluxcd.io/name\":\"core\",\"kustomize.toolkit.fluxcd.io/namespace\":\"flux-system\"},\"name\":\"prefer-dual-stack-admission-patch\",\"namespace\":\"prefer-dual-stack\"},\"spec\":{\"template\":{\"metadata\":{\"name\":\"prefer-dual-stack-admission-patch\"},\"spec\":{\"containers\":[{\"args\":[\"patch\",\"--webhook-name=prefer-dual-stack-admission\",\"--namespace=$(POD_NAMESPACE)\",\"--patch-validating=false\",\"--secret-name=prefer-dual-stack-admission\",\"--patch-failure-policy=Fail\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"docker.io/jettech/kube-webhook-certgen:v1.5.2\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"patch\"}],\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"prefer-dual-stack-admission\"}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"patch"}],"containers":[{"image":"docker.io/jettech/kube-webhook-certgen:v1.5.2","name":"patch"}]}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "prefer-dual-stack-admission-patch", Namespace: "prefer-dual-stack"
for: "08947f31-e8f4-40a9-a646-8de1de9b36bf.yaml": Job.batch "prefer-dual-stack-admission-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"prefer-dual-stack-admission-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"27aae0e2-98af-4159-a07b-73e447f0e410", "job-name":"prefer-dual-stack-admission-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"docker.io/jettech/kube-webhook-certgen:v1.5.2", Command:[]string(nil), Args:[]string{"patch", "--webhook-name=prefer-dual-stack-admission", "--namespace=$(POD_NAMESPACE)", "--patch-validating=false", "--secret-name=prefer-dual-stack-admission", "--patch-failure-policy=Fail"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0x4009228100)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0x400ab4a0b8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"prefer-dual-stack-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0x4021800b80), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutableDeploy.yaml: https://github.com/samip5/k8s-cluster/blob/main/cluster/core/prefer-dual-stack/deploy.yaml
tibordp
commented
3 years ago Hey - so, prefer-dual-stack-admission-create Job creates a new self-signed server private key and certificate for the webhook and stores it in prefer-dual-stack-admission secret. This is then used by the webhook server. The other job prefer-dual-stack-admission-patch takes the generated caBundle from the secret and patches the MutatingWebhookConfiguration with it.
Both Jobs are only meant to run once, after the manifest is applied as a convenience to avoid having to manually generate the certificate through some other means. They are not required after the secret is generated and can safely be deleted.
The way I read the logs you posted is that your CD system tries to patch the Job after it's already been created. Jobs are immutable, so this is expected to fail. That said, the initial logs of prefer-dual-stack-admission-create that you posted look good to me, it checks if secret already exists and if it doesn't creates a new one.
I think for your use case, manually creating the secret out of band would be a better setup. I'm sure it is somehow doable with cert-manager, but generally, the expected format is as follows:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: prefer-dual-stack-admission
namespace: prefer-dual-stack
data:
ca: <base64 encoded CA certificate>
cert: <base64 encoded server certificate>
key: <base64 encoded private key file>The server certificate should contain the following SANs (certificate can safely be self-signed, no need or benefit with a publicly trusted one):
prefer-dual-stack-webhookprefer-dual-stack-webhook.prefer-dual-stack.svcThe same CA bundle that is used to sign the server certificate should also be injected into the MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: prefer-dual-stack-admission
webhooks:
- name: prefer-dual-stack.tibordp.github.io
matchPolicy: Equivalent
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["services"]
scope: "Namespaced"
failurePolicy: Fail
sideEffects: None
admissionReviewVersions:
- v1
clientConfig:
caBundle: <same base64-encoded CA bundle as above>
service:
namespace: prefer-dual-stack
name: prefer-dual-stack-webhook
path: /mutate/servicesUnfortunately, I don't have non-letsencrypt issuer configured so self-signed certificates are not possible at the moment.
It's still not very documented.
EDIT: It has created it.
$ k describe secret prefer-dual-stack-admission -n prefer-dual-stack
Name: prefer-dual-stack-admission
Namespace: prefer-dual-stack
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
ca: 566 bytes
cert: 639 bytes
key: 227 bytesOK, I can certainly add the details to the doc, but with the secret present, what exactly is it that is not working?
samip5
commented
3 years ago Oh, never mind. It was just saying that it didin't see a secret so it created it. It seems to be working but the docs do need some improvement. :)
P.S Helm chart would be nice too if that's not too much to ask.
tibordp
commented
3 years ago Great to see that it worked in the end. I agree about the docs and I'll try to take a stab at the Helm chart too.
Will keep this issue open to track it.
What do secret do I need for it?