Closed jmreicha closed 4 years ago
I'd try some/all of the following for one of the applications:
kubectl port-forward
Feel also free to submit a support ticket so that we can take a closer look.
1). Nothing in the cert-manager logs, looks like timout/connection errors in the cert manager logs.
E1118 16:33:21.799398 1 controller.go:131] cert-manager/controller/ingress-shim "msg"="re-queuing item due to error processing" "error"="Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request" "key"="default/test"
2).
metrics-server
Name: metrics-server
Namespace: kube-system
Labels: kubernetes.io/cluster-service=true
kubernetes.io/name=Metrics-server
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Me...
Selector: k8s-app=metrics-server
Type: ClusterIP
IP: 10.245.47.213
Port: <unset> 443/TCP
TargetPort: main-port/TCP
Endpoints: 10.244.2.116:4443
Session Affinity: None
Events: <none>
cert-manager
Name: cert-manager-webhook
Namespace: cert-manager
Labels: app=webhook
app.kubernetes.io/instance=cert-manager
app.kubernetes.io/managed-by=Tiller
app.kubernetes.io/name=webhook
helm.sh/chart=cert-manager-v0.11.0
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"webhook","app.kubernetes.io/instance":"cert-manager","ap...
Selector: app.kubernetes.io/instance=cert-manager,app.kubernetes.io/managed-by=Tiller,app.kubernetes.io/name=webhook,app=webhook
Type: ClusterIP
IP: 10.245.173.138
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 10.244.3.27:6443
Session Affinity: None
Events: <none>
3 and 4). Can't seem to connect to these endpoints, they do respond, but I am getting a 403. Below is the response inside the cluster.
curl -k https://cert-manager-webhook.cert-manager
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
What port is the API server configured to listen on? I wonder if the port mappings are incorrect or if there is something blocking requests to the API server?
Adding hostNetwork: true
to the deployment spec "fixes" the issue but I'm not really sure why this would be needed.
The firewall rules in the DO console seem to indicate that the 10.0.0.0/8
network should be allowed, so I would have guessed that would include things in the Kubernetes cluster?
Hey @jmreicha. Sorry, this one fell off my radar.
Is this still an issue for you? If so, then I'd suggest to file a DO support ticket. That should kick off a process which is better suited to address customer support request in a reliable, short-term manner.
@timoreimann 👋
Still an issue, I already have a support ticket open. They said was better to use this issue 😄
@jmreicha did you manage to resolve the issue with our support, or am I misreading our internal communication?
@timoreimann Yep we got it sorted.
Just a note if anybody else comes across this issue, changing the cert-manager validating and mutating webhooks to failurePolicy: Ignore
as well as restarting the control plane seems to fix the issue.
Thanks for the note explaining how you got this fixed, appreciated.
After upgrading from 1.15.x to 1.16.0, it appears custom APIs seem to be broken. For example, running kubectl get apiservice shows these APIs to be unavailable.
Checking these APIs reveals more info.
The services above exist in the cluster, so I'm not sure what is happening. Any thoughts or ideas on how to fix this?