crictl logs broken (1.9)

[vrothberg]

crictl logs yields the following error unconditionally:

FATA[0000] The container has not set log path

We can reproduce this in each of our deployments and also following the minimal example in the docs. We looked inside the issue and it seems that crictl isn't able to get the container's logPath right (it's always left empty). I have tested it with crio 1.9, which is setting the log files accordingly (I can see them in the logs and access them from the host).

I've seen some fixes for critools 1.10 indicating that the issue has been fixed. If so, is there any chance we can backport them to 1.9?

Unrelated to the issue but following the backporting question: Is there a policy or process established for critools to decide which commits should be backported? We're trying to roll fast and will update to K8s 1.10 in the coming months, but we usually need a few months of time.

[vrothberg]

Unrelated to the issue but following the backporting question: Is there a policy or process established for critools to decide which commits should be backported? We're trying to roll fast and will update to K8s 1.10 in the coming months, but we usually need a few months of time.

We gladly jump in to backport things if that would help.

[Random-Liu]

I've seen some fixes for critools 1.10 indicating that the issue has been fixed. If so, is there any chance we can backport them to 1.9?

Which fix are you talking about? It seems to be a crio bug to me, which doesn't return LogPath in ContainerStatus.

[vrothberg]

Which fix are you talking about? It seems to be a crio bug to me, which doesn't return LogPath in ContainerStatus.

@Random-Liu I skimmed the history and saw a few commits related to logs. I am not sure if it's a crio bug as it works with kubectl logs.

[feiskyer]

@vrothberg Could you show the output of crictl inspect <container-id>?

[feiskyer]

By the way, with master branch, logs won't work with current examples. This is because logPath and logDirectory is not set. Opened #321 to reflect recent changes.

[vrothberg]

I took some more time looking into the issue and realized that logs work when using docker, so I begin to doubt my hypothesis that it's a crictl issue.

@feiskyer Below are the logs (when using CRI-O). The logPath is clearly empty which explains the error, but I am not sure why:

{
  "status": {
    "id": "e7afd83de2dd789e2d70958b7d9a18a42fddf06d590edab32690fb053e55ac10",
    "metadata": {
      "attempt": 0,
      "name": "dex"
    },
    "state": "CONTAINER_RUNNING",
    "createdAt": "2018-06-05T10:45:45.750328318Z",
    "startedAt": "2018-06-05T10:45:46.285758221Z",
    "finishedAt": "1970-01-01T00:00:00Z",
    "exitCode": 0,
    "image": {
      "image": "docker.io/sles12/caasp-dex:2.7.1"
    },
    "imageRef": "docker.io/sles12/caasp-dex@sha256:4997cc886a0c3c103588e5c4195725513bd4d6ab2e758764af00998dfe25c1bb",
    "reason": "",
    "message": "",
    "labels": {
      "io.kubernetes.container.name": "dex",
      "io.kubernetes.pod.name": "dex-7bcd9b8cb5-9bj5n",
      "io.kubernetes.pod.namespace": "kube-system",
      "io.kubernetes.pod.uid": "93c031ee-68ad-11e8-a946-5afab06adeee"
    },
    "annotations": {
      "io.kubernetes.container.hash": "5eea8f3d",
      "io.kubernetes.container.ports": "[{\"name\":\"https\",\"containerPort\":5556,\"protocol\":\"TCP\"}]",
      "io.kubernetes.container.restartCount": "0",
      "io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
      "io.kubernetes.container.terminationMessagePolicy": "File",
      "io.kubernetes.pod.terminationGracePeriod": "30"
    },
    "mounts": [
      {
        "containerPath": "/etc/dex/cfg",
        "hostPath": "/var/lib/kubelet/pods/93c031ee-68ad-11e8-a946-5afab06adeee/volumes/kubernetes.io~configmap/config",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": false
      },
      {
        "containerPath": "/etc/dex/tls",
        "hostPath": "/var/lib/kubelet/pods/93c031ee-68ad-11e8-a946-5afab06adeee/volumes/kubernetes.io~secret/tls",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": false
      },
      {
        "containerPath": "/etc/pki/trust/anchors/SUSE_CaaSP_CA.crt",
        "hostPath": "/etc/pki/trust/anchors/SUSE_CaaSP_CA.crt",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": false
      },
      {
        "containerPath": "/var/run/secrets/kubernetes.io/serviceaccount",
        "hostPath": "/var/lib/kubelet/pods/93c031ee-68ad-11e8-a946-5afab06adeee/volumes/kubernetes.io~secret/dex-token-qrlk5",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": true,
        "selinuxRelabel": false
      },
      {
        "containerPath": "/etc/hosts",
        "hostPath": "/var/lib/kubelet/pods/93c031ee-68ad-11e8-a946-5afab06adeee/etc-hosts",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": false
      },
      {
        "containerPath": "/dev/termination-log",
        "hostPath": "/var/lib/kubelet/pods/93c031ee-68ad-11e8-a946-5afab06adeee/containers/dex/0acacc1a",
        "propagation": "PROPAGATION_PRIVATE",
        "readonly": false,
        "selinuxRelabel": false
      }
    ],
    "logPath": ""
  }
}

[vrothberg]

Closing the issue as it turns out to be a crio issue: https://github.com/kubernetes-incubator/cri-o/pull/1591#issuecomment-394734679

[LinTechSo]

Hi, any updates ?

[haircommander]

this should be fixed, and 1.9 is out of support. are you encountering this in a newer version?

[LinTechSo]

Hi @haircommander . thanks for the reply Yes. i have opened issue about this. https://github.com/kubernetes-sigs/cri-tools/issues/1078