search

tintoy / dotnet-kube-client

A Kubernetes API client for .NET Standard / .NET Core
MIT License
192 stars 33 forks source link

Int32OrStringV1 throwing exception on deployment liveness probe #70

Closed jonstelly closed 5 years ago

jonstelly commented 5 years ago

I'm using KubeClient to list deployments and getting an exception when processing a liveness probe.

KubeClient version: 2.2.7

Kubernetes Version info:

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

The exception looks like this:

Newtonsoft.Json.JsonReaderException : Could not convert string to integer: scheme. Path 'items[0].spec.template.spec.containers[0].livenessProbe.httpGet.port', line 1, position 1875.
   at Newtonsoft.Json.JsonReader.ReadInt32String(String s)
   at Newtonsoft.Json.JsonTextReader.FinishReadQuotedNumber(ReadType readType)
   at Newtonsoft.Json.JsonTextReader.ReadAsInt32()
   at KubeClient.Models.Converters.Int32OrStringV1Converter.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in ...\dotnet-kube-client\src\KubeClient\Models\Converters\Int32OrStringV1Converter.cs:line 75

From debugging it seems to be the liveness check for nginx ingress controller (deployed with Helm from here):

kubectl describe deployment/ingress-private-nginx-ingress-controller

Name:                   ingress-private-nginx-ingress-controller
Namespace:              default
CreationTimestamp:      Mon, 04 Mar 2019 21:49:01 -0600
Labels:                 app=nginx-ingress
                        chart=nginx-ingress-1.1.4
                        component=controller
                        heritage=Tiller
                        release=ingress-private
Annotations:            deployment.kubernetes.io/revision: 2
Selector:               app=nginx-ingress,component=controller,release=ingress-private
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app=nginx-ingress
                    component=controller
                    release=ingress-private
  Service Account:  ingress-private-nginx-ingress
  Containers:
   nginx-ingress-controller:
    Image:       quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
    Ports:       80/TCP, 443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=default/ingress-private-nginx-ingress-default-backend
      --election-id=ingress-controller-leader
      --ingress-class=private
      --configmap=default/ingress-private-nginx-ingress-controller
      --tcp-services-configmap=default/ingress-private-nginx-ingress-tcp
      --enable-ssl-passthrough=true
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:           <none>
  Volumes:            <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-private-nginx-ingress-controller-5775b9579f (1/1 replicas created)
Events:          <none>

the liveness port for 10254 is where the exception happens. I was able to capture the JSON for this deployment by replacing KubeResourceClient.cs line 178 with var json = await responseMessage.Content.ReadAsStringAsync();

{
  "kind": "DeploymentList",
  "apiVersion": "apps/v1",
  "metadata": { "selfLink": "/apis/apps/v1/namespaces/default/deployments", "resourceVersion": "1959501" },
  "items": [
    {
      "metadata": {
        "name": "ingress-private-nginx-ingress-controller",
        "namespace": "default",
        "selfLink": "/apis/apps/v1/namespaces/default/deployments/ingress-private-nginx-ingress-controller",
        "uid": "9d22afb8-3ef9-11e9-aa7e-00155d10f745",
        "resourceVersion": "1258855",
        "generation": 2,
        "creationTimestamp": "2019-03-05T03:49:01Z",
        "labels": { "app": "nginx-ingress", "chart": "nginx-ingress-1.1.4", "component": "controller", "heritage": "Tiller", "release": "ingress-private" },
        "annotations": { "deployment.kubernetes.io/revision": "2" }
      },
      "spec": {
        "replicas": 1,
        "selector": { "matchLabels": { "app": "nginx-ingress", "component": "controller", "release": "ingress-private" } },
        "template": {
          "metadata": { "creationTimestamp": null, "labels": { "app": "nginx-ingress", "component": "controller", "release": "ingress-private" } },
          "spec": {
            "containers": [
              {
                "name": "nginx-ingress-controller",
                "image": "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0",
                "args": [
                  "/nginx-ingress-controller",
                  "--default-backend-service=default/ingress-private-nginx-ingress-default-backend",
                  "--election-id=ingress-controller-leader",
                  "--ingress-class=private",
                  "--configmap=default/ingress-private-nginx-ingress-controller",
                  "--tcp-services-configmap=default/ingress-private-nginx-ingress-tcp",
                  "--enable-ssl-passthrough=true"
                ],
                "ports": [
                  { "name": "http", "containerPort": 80, "protocol": "TCP" },
                  { "name": "https", "containerPort": 443, "protocol": "TCP" }
                ],
                "env": [
                  { "name": "POD_NAME", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.name" } } },
                  { "name": "POD_NAMESPACE", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.namespace" } } }
                ],
                "resources": {},
                "livenessProbe": {
                  "httpGet": { "path": "/healthz", "port": 10254, "scheme": "HTTP" },
                  "initialDelaySeconds": 10,
                  "timeoutSeconds": 1,
                  "periodSeconds": 10,
                  "successThreshold": 1,
                  "failureThreshold": 3
                },
                "readinessProbe": {
                  "httpGet": { "path": "/healthz", "port": 10254, "scheme": "HTTP" },
                  "initialDelaySeconds": 10,
                  "timeoutSeconds": 1,
                  "periodSeconds": 10,
                  "successThreshold": 1,
                  "failureThreshold": 3
                },
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",
                "imagePullPolicy": "IfNotPresent",
                "securityContext": { "capabilities": { "add": ["NET_BIND_SERVICE"], "drop": ["ALL"] }, "runAsUser": 33, "procMount": "Default" }
              }
            ],
            "restartPolicy": "Always",
            "terminationGracePeriodSeconds": 60,
            "dnsPolicy": "ClusterFirst",
            "serviceAccountName": "ingress-private-nginx-ingress",
            "serviceAccount": "ingress-private-nginx-ingress",
            "securityContext": {},
            "schedulerName": "default-scheduler"
          }
        },
        "strategy": { "type": "RollingUpdate", "rollingUpdate": { "maxUnavailable": 1, "maxSurge": 1 } },
        "revisionHistoryLimit": 10,
        "progressDeadlineSeconds": 2147483647
      },
      "status": {
        "observedGeneration": 2,
        "replicas": 1,
        "updatedReplicas": 1,
        "readyReplicas": 1,
        "availableReplicas": 1,
        "conditions": [
          {
            "type": "Available",
            "status": "True",
            "lastUpdateTime": "2019-03-05T03:49:01Z",
            "lastTransitionTime": "2019-03-05T03:49:01Z",
            "reason": "MinimumReplicasAvailable",
            "message": "Deployment has minimum availability."
          }
        ]
      }
    }
  ]
}
tintoy commented 5 years ago

Hi! Sorry about that, I guess I was so focused on the serialisation that deserialisation dropped off the radar somewhat. Thanks for the fix - I'll publish a new version shortly.

jonstelly commented 5 years ago

Not a problem, I appreciate your work on this library.

tintoy commented 5 years ago

Ok, published v2.2.8 to NuGet. It should show up in 15-20 minutes (depending on how busy they are), but you should be able to fetch it from the MyGet feed if you don't want to wait :)