Container label sometimes only contains sha256

sylr commented 5 years ago

Hi,

I have a few containers that have metrics where the image label only contains the sha256, e.g.:

container_cpu_user_seconds_total{
    agentpool="agentzone1",
    beta_kubernetes_io_arch="amd64",
    beta_kubernetes_io_instance_type="Standard_D4s_v3",
    beta_kubernetes_io_os="linux",
    container_name="tiller",
    failure_domain_beta_kubernetes_io_region="westeurope",
    failure_domain_beta_kubernetes_io_zone="westeurope-1",
    id="/kubepods/burstable/podd8270dc0-513e-11e9-8605-000d3a21338b/bc962a0460ddf51eb591b0f2132c3a3c6801062c69e997a30e26398704b53406",
->  image="sha256:9c3111f27796ea54c7791ada8d4733260aef65000efdb32e90848141b5c67827",
    instance="xxxxxxx-agentzone1-vmss-00000n",
    job="kubernetes-nodes-cadvisor",
    kubernetes_azure_com_cluster="xxxxxxxxxxxxx",
    kubernetes_io_hostname="xxxxxxx-agentzone1-vmss-00000n",
    kubernetes_io_role="agent",
    name="k8s_tiller_tiller-deploy-c4665b6cb-ff5h7_iic_d8270dc0-513e-11e9-8605-000d3a21338b_0",
    namespace="iic",
    pod_name="tiller-deploy-c4665b6cb-ff5h7",
    storageprofile="managed",
    storagetier="Premium_LRS"
}

vs

container_cpu_user_seconds_total{
    agentpool="agentzone1",
    beta_kubernetes_io_arch="amd64",
    beta_kubernetes_io_instance_type="Standard_D4s_v3",
    beta_kubernetes_io_os="linux",
    container_name="ubuntu",
    failure_domain_beta_kubernetes_io_region="westeurope",
    failure_domain_beta_kubernetes_io_zone="westeurope-1",
    id="/kubepods/burstable/pod88b2220a-50c9-11e9-8605-000d3a21338b/a960a59adb13b5116da85b64d255887e820e859185a9797453bbe35c67fbba0f",
->  image="quay.io/sylr/ubuntu@sha256:8a66a8690b659381f96c70c13003d720695e72cda241af8998033ef14af0697c",
    instance="xxxxxxx-agentzone1-vmss-00000n",
    job="kubernetes-nodes-cadvisor",
    kubernetes_azure_com_cluster="xxxxxxxxxxxxx",
    kubernetes_io_hostname="xxxxxxx-agentzone1-vmss-00000n",
    kubernetes_io_role="agent",
    name="k8s_ubuntu_ubuntu-mn58x_default_88b2220a-50c9-11e9-8605-000d3a21338b_0",
    namespace="default",
    pod_name="ubuntu-mn58x",
    storageprofile="managed",
    storagetier="Premium_LRS"
}

I've been having this problem since kubernetes 1.10.x at least, I'm currently using 1.13.5.

Those metrics comes from kubernetes.default.svc:443/api/v1/nodes/${1}/proxy/metrics/cadvisor

sylr commented 5 years ago

/assign @dashpole

dashpole commented 5 years ago

It looks like cAdvisor gets its image from docker directly: https://github.com/google/cadvisor/blob/master/container/docker/handler.go#L221.

Can you see what docker ps returns for each?

google / cadvisor

Container label sometimes only contains sha256 #2208