search

containerd / cri

Moved to https://github.com/containerd/containerd/tree/master/pkg/cri . If you wish to submit issues/PRs, please submit to https://github.com/containerd/containerd
https://github.com/containerd/containerd/tree/master/pkg/cri
Apache License 2.0
900 stars 349 forks source link

Deadlock after containerd restart #1018

Closed steven-sheehy closed 5 years ago

steven-sheehy commented 5 years ago

After restarting containerd via sudo systemctl restart containerd, I have pods stuck in various state (Terminating,Init, Running 0/1) indefinitely. Also, containerd is running but printing no logs and seems to be deadlocked. crictl is not able to connect to socket.

Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 243351 (keepalived) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 166921 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 167372 (kube-proxy) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 95300 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 95561 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 95708 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96032 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96251 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96454 (sh) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96637 (sfacctd) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96638 (sfacctd) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96639 (sfacctd) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96640 (sfacctd) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96489 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: containerd.service: Found left-over process 96561 (pause) in control group while starting unit. Ignoring.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 03 19:54:41 qaedge1fce1-2 systemd[1]: Started containerd container runtime.
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.036086126Z" level=info msg="starting containerd" revision=9b32062dc1f5a7c2564315c269b5059754f12b9d version=v1.2.1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.036504273Z" level=info msg="loading plugin "io.containerd.content.v1.content"..." type=io.containerd.content.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.036543269Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." type=io.containerd.snapshotter.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.037281046Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.037302772Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.aufs"..." type=io.containerd.snapshotter.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.038951230Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.native"..." type=io.containerd.snapshotter.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.039034557Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.039177918Z" level=info msg="loading plugin "io.containerd.snapshotter.v1.zfs"..." type=io.containerd.snapshotter.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040123368Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.zfs" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040157657Z" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." type=io.containerd.metadata.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040187729Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040201323Z" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040558180Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040587066Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040645457Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040688366Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040714870Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040734934Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040760259Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040781371Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040800431Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040819297Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.040890718Z" level=debug msg="loading tasks in namespace" namespace=k8s.io
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.099621328Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.099740806Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.100894820Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.102886520Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.102967179Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.102992443Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103011922Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103030741Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103049166Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103068414Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103087174Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103105195Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103123540Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103197279Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103222992Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103243233Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103262338Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103366222Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntime:{Type:io.containerd.runtime.v1.linux Engine: Root: Options:<nil>} UntrustedWorkloadRuntime:{Type: Engine: Root: Options:<nil>} Runtimes:map[] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Auths:map[]} StreamServerAddress:127.0.0.1 StreamServerPort:0 EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103395529Z" level=info msg="Connect containerd service"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.103576378Z" level=info msg="Get image filesystem path "/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs""
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104034667Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104169190Z" level=info msg="Start subscribing containerd event"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104248653Z" level=info msg=serving... address="/run/containerd/debug.sock"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104255406Z" level=info msg="Start recovering state"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104332419Z" level=info msg=serving... address="/run/containerd/containerd.sock"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.104353186Z" level=info msg="containerd successfully booted in 0.069395s"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.172187227Z" level=debug msg="garbage collected" d=31.322746ms
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.181128154Z" level=debug msg="Loaded sandbox {Metadata:{ID:10b309d5d2945eee4cabe23346a56ab3e7c037a1ff21e6d5646e98b5d3fdfbf0 Name:node-exporter-85752_production_4a73066e-d640-11e8-a966-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:node-exporter-85752,Uid:4a73066e-d640-11e8-a966-00505691e4af,Namespace:production,Attempt:0,},Hostname:node-exporter-85752,LogDirectory:/var/log/pods/4a73066e-d640-11e8-a966-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:9100,HostPort:9100,HostIp:,}],Labels:map[string]string{app: prometheus,component: node-exporter,controller-revision-hash: 92065873,io.kubernetes.pod.name: node-exporter-85752,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 4a73066e-d640-11e8-a966-00505691e4af,pod-template-generation: 3,release: edge,},Annotations:map[string]string{kubernetes.io/config.seen: 2018-12-03T03:27:57.45340215Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod4a73066e-d640-11e8-a966-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-2aadbbd3-45fb-6480-b792-9b3c15af0a04 IP:10.214.128.36 RuntimeHandler:} Status:0xc4202c6040 Container:0xc4202c5be0 NetNS:0xc42062c010 StopCh:0xc42026e020}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.256186652Z" level=debug msg="Loaded sandbox {Metadata:{ID:14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841 Name:local-storage-759d6_production_6a6f0656-d640-11e8-a966-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:local-storage-759d6,Uid:6a6f0656-d640-11e8-a966-00505691e4af,Namespace:production,Attempt:0,},Hostname:local-storage-759d6,LogDirectory:/var/log/pods/6a6f0656-d640-11e8-a966-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[],Labels:map[string]string{app: local-storage,controller-revision-hash: 1049966276,io.kubernetes.pod.name: local-storage-759d6,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 6a6f0656-d640-11e8-a966-00505691e4af,pod-template-generation: 3,},Annotations:map[string]string{kubernetes.io/config.seen: 2018-12-03T03:27:57.453413179Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:true,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-0daa055b-e807-7690-9eb9-a4578b3030af IP:10.214.128.39 RuntimeHandler:} Status:0xc42025c380 Container:0xc4202c5c00 NetNS:0xc4206beb70 StopCh:0xc4202e2140}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.312499335Z" level=debug msg="Loaded sandbox {Metadata:{ID:264a6a17b38e19a5bcc5c7b18b078efe5b103a596d4340a67bf18cb16d77bc41 Name:edge-sfacctd-54bbb77548-brsw6_production_00f37025-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:edge-sfacctd-54bbb77548-brsw6,Uid:00f37025-0ecc-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:edge-sfacctd-54bbb77548-brsw6,LogDirectory:/var/log/pods/00f37025-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:UDP,ContainerPort:6343,HostPort:0,HostIp:,}],Labels:map[string]string{app: edge-sfacctd,io.kubernetes.pod.name: edge-sfacctd-54bbb77548-brsw6,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 00f37025-0ecc-11e9-ac99-00505691e4af,pod-template-hash: 1066633104,},Annotations:map[string]string{config-file-hash: -860870525,kubernetes.io/config.seen: 2019-01-02T20:23:14.107988059Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod00f37025-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-eb632545-e00c-cd08-b808-660b0d59d3b0 IP:10.214.128.41 RuntimeHandler:} Status:0xc42025cf00 Container:0xc4202c5c40 NetNS:0xc42003ba70 StopCh:0xc4202e2e20}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.373165469Z" level=debug msg="Loaded sandbox {Metadata:{ID:3963deae9861bd99f1fde35819ecbf8642098e03a742bd1fab7f316495645f85 Name:grafana-c498c86c9-vnpr2_production_00d4e816-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:grafana-c498c86c9-vnpr2,Uid:00d4e816-0ecc-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:grafana-c498c86c9-vnpr2,LogDirectory:/var/log/pods/00d4e816-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:9001,HostPort:0,HostIp:,} &PortMapping{Protocol:TCP,ContainerPort:3000,HostPort:0,HostIp:,}],Labels:map[string]string{app: grafana,io.kubernetes.pod.name: grafana-c498c86c9-vnpr2,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 00d4e816-0ecc-11e9-ac99-00505691e4af,pod-template-hash: 705474275,release: edge,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.104688094Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod00d4e816-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:&Int64Value{Value:472,},ReadonlyRootfs:false,SupplementalGroups:[472],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-1294623c-f01a-334f-ebcb-b8a44e9c8432 IP:10.214.128.37 RuntimeHandler:} Status:0xc42025da40 Container:0xc4202c5c60 NetNS:0xc42003bf10 StopCh:0xc4202e3b60}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.428304608Z" level=debug msg="Loaded sandbox {Metadata:{ID:49539e8ac5a735c8946d683198957962ec0e186a9774541d8bb7c3fae49342ef Name:weave-net-npqrq_kube-system_d6448f2a-e152-11e8-852a-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:weave-net-npqrq,Uid:d6448f2a-e152-11e8-852a-00505691e4af,Namespace:kube-system,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/d6448f2a-e152-11e8-852a-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.0.22.45 10.0.22.46],Searches:[firescope.int firescope.com],Options:[],},PortMappings:[],Labels:map[string]string{controller-revision-hash: 3091379672,io.kubernetes.pod.name: weave-net-npqrq,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: d6448f2a-e152-11e8-852a-00505691e4af,name: weave-net,pod-template-generation: 2,},Annotations:map[string]string{kubernetes.io/config.seen: 2018-12-03T03:27:57.453409854Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/podd6448f2a-e152-11e8-852a-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:NODE,Ipc:POD,},SelinuxOptions:&SELinuxOption{User:,Role:,Type:,Level:,},RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:true,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath: IP: RuntimeHandler:} Status:0xc420700540 Container:0xc4202c5c80 NetNS:<nil> StopCh:0xc4206fe7a0}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.488710914Z" level=debug msg="Loaded sandbox {Metadata:{ID:616dcf6c787c956a34f93f19f907d0d59a1b9d003afed92a2ca86bf327c54137 Name:edge-master-5f55799bc-v8tb2_production_00aec7c7-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:edge-master-5f55799bc-v8tb2,Uid:00aec7c7-0ecc-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:edge-master-5f55799bc-v8tb2,LogDirectory:/var/log/pods/00aec7c7-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:8080,HostPort:0,HostIp:,}],Labels:map[string]string{app: edge-master,io.kubernetes.pod.name: edge-master-5f55799bc-v8tb2,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 00aec7c7-0ecc-11e9-ac99-00505691e4af,pod-template-hash: 191135567,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.097059867Z,kubernetes.io/config.source: api,prometheus.io/port: 8080,prometheus.io/scrape: true,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/pod00aec7c7-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-beb01506-0843-20ef-4001-80cfd2597b92 IP:10.214.128.40 RuntimeHandler:} Status:0xc420701040 Container:0xc4202c5ca0 NetNS:0xc42077d100 StopCh:0xc4206ff420}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.545147897Z" level=debug msg="Loaded sandbox {Metadata:{ID:7420d884cc2262138fcdb86f9e7e504d3ea9f248c6696fc24fad054b2bdd8388 Name:keepalived-wnk56_production_e58eabdc-f6ad-11e8-9d1d-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:keepalived-wnk56,Uid:e58eabdc-f6ad-11e8-9d1d-00505691e4af,Namespace:production,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/e58eabdc-f6ad-11e8-9d1d-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.0.22.45 10.0.22.46],Searches:[firescope.int firescope.com],Options:[],},PortMappings:[],Labels:map[string]string{app: keepalived,controller-revision-hash: 1825412473,io.kubernetes.pod.name: keepalived-wnk56,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: e58eabdc-f6ad-11e8-9d1d-00505691e4af,pod-template-generation: 4,release: edge,role: network,},Annotations:map[string]string{checksum/config: 91d37c5456226df939a52d6888299901424cdc99af60de6477131434e02fd11f,kubernetes.io/config.seen: 2018-12-03T03:45:37.596703812Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/pode58eabdc-f6ad-11e8-9d1d-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:true,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath: IP: RuntimeHandler:} Status:0xc420701b40 Container:0xc4202c5cc0 NetNS:<nil> StopCh:0xc42071c060}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.605038825Z" level=debug msg="Loaded sandbox {Metadata:{ID:8b2db17fb0c9ecc79e54ae5dbd563fe9969ea11e6ce33af52eb684fa15a83b07 Name:kube-proxy-6wbdx_kube-system_5672db15-0ec3-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:kube-proxy-6wbdx,Uid:5672db15-0ec3-11e9-ac99-00505691e4af,Namespace:kube-system,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/5672db15-0ec3-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.0.22.45 10.0.22.46],Searches:[firescope.int firescope.com],Options:[],},PortMappings:[],Labels:map[string]string{controller-revision-hash: 4270652902,io.kubernetes.pod.name: kube-proxy-6wbdx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 5672db15-0ec3-11e9-ac99-00505691e4af,k8s-app: kube-proxy,pod-template-generation: 3,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T19:19:34.337599265Z,kubernetes.io/config.source: api,scheduler.alpha.kubernetes.io/critical-pod: ,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/besteffort/pod5672db15-0ec3-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:true,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath: IP: RuntimeHandler:} Status:0xc420720600 Container:0xc4202c5ce0 NetNS:<nil> StopCh:0xc42071cc20}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.666172121Z" level=debug msg="Loaded sandbox {Metadata:{ID:9d0374d05c5de01b86fb620c846b9878dd17c8ccfb6e233a847e6e373b39b362 Name:edge-master-5f55799bc-ljc2b_production_0058524c-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:edge-master-5f55799bc-ljc2b,Uid:0058524c-0ecc-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:edge-master-5f55799bc-ljc2b,LogDirectory:/var/log/pods/0058524c-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:5],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:8080,HostPort:0,HostIp:,}],Labels:map[string]string{app: edge-master,io.kubernetes.pod.name: edge-master-5f55799bc-ljc2b,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 0058524c-0ecc-11e9-ac99-00505691e4af,pod-template-hash: 191135567,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.084037614Z,kubernetes.io/config.source: api,prometheus.io/port: 8080,prometheus.io/scrape: true,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/pod0058524c-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-7d6d4902-fa1a-d9d7-77ac-4d8d36e090c4 IP:10.214.128.38 RuntimeHandler:} Status:0xc420721100 Container:0xc4202c5d00 NetNS:0xc42077dd90 StopCh:0xc42071d880}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.724488701Z" level=debug msg="Loaded sandbox {Metadata:{ID:e22555ea5e47081df5b440839b1bfeeeaf89fa5f454bb4ea4db36d9b4be48fa1 Name:egress-c9pqw_production_91d190c5-e6be-11e8-9cd7-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:egress-c9pqw,Uid:91d190c5-e6be-11e8-9cd7-00505691e4af,Namespace:production,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/91d190c5-e6be-11e8-9cd7-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.0.22.45 10.0.22.46],Searches:[firescope.int firescope.com],Options:[],},PortMappings:[],Labels:map[string]string{app: egress,controller-revision-hash: 3142929071,io.kubernetes.pod.name: egress-c9pqw,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 91d190c5-e6be-11e8-9cd7-00505691e4af,pod-template-generation: 4,},Annotations:map[string]string{kubernetes.io/config.seen: 2018-12-03T03:27:57.453373898Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/pod91d190c5-e6be-11e8-9cd7-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:true,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath: IP: RuntimeHandler:} Status:0xc420721c00 Container:0xc4202c5d20 NetNS:<nil> StopCh:0xc4207d0460}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.790395728Z" level=debug msg="Loaded sandbox {Metadata:{ID:e2d1a384e7b31bfca54d60ee1e7829699a5251fc0ae3cfc1794072f0183f8533 Name:edge-worker-858ff6ff7f-9k978_production_00889b21-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:edge-worker-858ff6ff7f-9k978,Uid:00889b21-0ecc-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:edge-worker-858ff6ff7f-9k978,LogDirectory:/var/log/pods/00889b21-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[single-request-reopen ndots:1],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:8081,HostPort:0,HostIp:,}],Labels:map[string]string{app: edge-worker,io.kubernetes.pod.name: edge-worker-858ff6ff7f-9k978,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 00889b21-0ecc-11e9-ac99-00505691e4af,pod-template-hash: 4149929939,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.084671118Z,kubernetes.io/config.source: api,prometheus.io/port: 8081,prometheus.io/scrape: true,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod00889b21-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-e935c9ae-66e5-b34a-bb25-e9fc4e226ccd IP:10.214.128.42 RuntimeHandler:} Status:0xc4202c67c0 Container:0xc4202c5d40 NetNS:0xc42062cae0 StopCh:0xc42026e980}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.849944318Z" level=debug msg="Loaded sandbox {Metadata:{ID:fab4e9c2bcadf1a652fe840cdc4a87c46d08e856ce3568bb8ed9e7f2c18ed350 Name:edge-inventory-669b6ff887-qtm5z_production_4f103bac-0ec4-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:edge-inventory-669b6ff887-qtm5z,Uid:4f103bac-0ec4-11e9-ac99-00505691e4af,Namespace:production,Attempt:0,},Hostname:,LogDirectory:/var/log/pods/4f103bac-0ec4-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.96.0.10],Searches:[production.svc.cluster.local svc.cluster.local cluster.local firescope.int firescope.com],Options:[ndots:1 single-request-reopen],},PortMappings:[&PortMapping{Protocol:TCP,ContainerPort:8084,HostPort:8084,HostIp:,}],Labels:map[string]string{app: edge-inventory,io.kubernetes.pod.name: edge-inventory-669b6ff887-qtm5z,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 4f103bac-0ec4-11e9-ac99-00505691e4af,pod-template-hash: 2256299443,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.055686266Z,kubernetes.io/config.source: api,prometheus.io/port: 8084,prometheus.io/scrape: true,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod4f103bac-0ec4-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath: IP: RuntimeHandler:} Status:0xc4202c7340 Container:0xc4202c5d60 NetNS:<nil> StopCh:0xc42026f640}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.905052025Z" level=debug msg="Loaded sandbox {Metadata:{ID:fbd8d3488cb4fb18706df86d1aa94c2a2fff737b16ffb44062576191c974f1cf Name:coredns-78fcdf6894-fvzzb_kube-system_005a726f-0ecc-11e9-ac99-00505691e4af_0 Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:coredns-78fcdf6894-fvzzb,Uid:005a726f-0ecc-11e9-ac99-00505691e4af,Namespace:kube-system,Attempt:0,},Hostname:coredns-78fcdf6894-fvzzb,LogDirectory:/var/log/pods/005a726f-0ecc-11e9-ac99-00505691e4af,DnsConfig:&DNSConfig{Servers:[10.0.22.45 10.0.22.46],Searches:[firescope.int firescope.com],Options:[],},PortMappings:[&PortMapping{Protocol:UDP,ContainerPort:53,HostPort:0,HostIp:,} &PortMapping{Protocol:TCP,ContainerPort:53,HostPort:0,HostIp:,} &PortMapping{Protocol:TCP,ContainerPort:9153,HostPort:0,HostIp:,}],Labels:map[string]string{io.kubernetes.pod.name: coredns-78fcdf6894-fvzzb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 005a726f-0ecc-11e9-ac99-00505691e4af,k8s-app: kube-dns,pod-template-hash: 3497892450,},Annotations:map[string]string{kubernetes.io/config.seen: 2019-01-02T20:23:14.084417815Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:/kubepods/burstable/pod005a726f-0ecc-11e9-ac99-00505691e4af,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:,RunAsGroup:nil,},Sysctls:map[string]string{},},} NetNSPath:/var/run/netns/cni-190c60d6-8745-85d3-7492-092fbd33c786 IP:10.214.128.14 RuntimeHandler:} Status:0xc4202c7e40 Container:0xc4202c5d80 NetNS:0xc42062d630 StopCh:0xc4207fa2a0}"
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.941929068Z" level=debug msg="Start writing stream "stdout" to log file "/var/log/pods/00f37025-0ecc-11e9-ac99-00505691e4af/edge-sfacctd/0.log""
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.941993798Z" level=debug msg="Start writing stream "stderr" to log file "/var/log/pods/00f37025-0ecc-11e9-ac99-00505691e4af/edge-sfacctd/0.log""
Jan 03 19:54:42 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:42.976085507Z" level=debug msg="Loaded container {Metadata:{ID:013d12bc9156460a60795ef4d571dba4c96a57e3ae2f3b76beec57cacd604727 Name:edge-sfacctd_edge-sfacctd-54bbb77548-brsw6_production_00f37025-0ecc-11e9-ac99-00505691e4af_0 SandboxID:264a6a17b38e19a5bcc5c7b18b078efe5b103a596d4340a67bf18cb16d77bc41 Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:edge-sfacctd,Attempt:0,},Image:&ImageSpec{Image:sha256:b1d42fde5d0ac7eb628a6e607e8a9ba2af01d657afb3ae653932d755a87f2c01,},Command:[/bin/sh -c],Args:[sfacctd -f /etc/pmacct/sflow_replicator.conf -F /var/run/pmacct/sfacctd_tee.pid; sfacctd -f /etc/pmacct/sflow_receiver.conf -F /var/run/pmacct/sfacctd.pid],WorkingDir:,Envs:[&KeyValue{Key:EDGE_EMI_PORT_8004_TCP,Value:tcp://10.105.73.122:8004,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_AMQP,Value:5672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP,Value:tcp://10.102.76.76:4369,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP,Value:tcp://10.107.185.128:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP,Value:tcp://10.102.76.76:5672,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP,Value:tcp://10.98.195.79:9000,} &KeyValue{Key:GRAFANA_PUBLIC_SERVICE_PORT,Value:9001,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP,Value:tcp://10.111.206.5:443,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_PORT,Value:2055,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP,Value:tcp://10.109.91.93:9002,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP,Value:tcp://10.104.141.5:9002,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:KUBERNETES_SERVICE_PORT,Value:443,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_PORT,Value:8043,} &KeyValue{Key:EDGE_EMI_PORT,Value:tcp://10.105.73.122:8004,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_HOST,Value:10.109.2.52,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT,Value:tcp://10.98.136.14:27017,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_NETFLOW1,Value:2055,} &KeyValue{Key:EDGE_NFACCTD_PORT,Value:udp://10.102.19.83:2055,} &KeyValue{Key:KUBERNETES_PORT,Value:tcp://10.96.0.1:443,} &KeyValue{Key:GRAFANA_PUBLIC_SERVICE_HOST,Value:10.104.165.161,} &KeyValue{Key:GRAFANA_PUBLIC_PORT,Value:tcp://10.104.165.161:9001,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_PORT,Value:27017,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_ADDR,Value:10.109.208.119,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_HOST,Value:10.107.185.128,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT,Value:tcp://10.99.30.144:2200,} &KeyValue{Key:GRAFANA_SERVICE_PORT_SERVICE,Value:9001,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP,Value:tcp://10.102.76.76:9090,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_PORT,Value:8004,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP,Value:tcp://10.104.136.1:9000,} &KeyValue{Key:PROMETHEUS_ADAPTER_SERVICE_PORT,Value:443,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP,Value:udp://10.102.19.83:2055,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_ADDR,Value:10.110.5.244,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_ADDR,Value:10.105.73.122,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_PORT,Value:9002,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT,Value:162,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_NETFLOW2,Value:2100,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_SERVICE_PORT,Value:9002,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT,Value:tcp://10.104.141.5:9002,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT,Value:15672,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:KUBERNETES_PORT_443_TCP,Value:tcp://10.96.0.1:443,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_PROTO,Value:tcp,} &KeyValue{Key:DASHBOARD_SERVICE_HOST,Value:10.104.136.1,} &KeyValue{Key:PROMETHEUS_ADAPTER_SERVICE_HOST,Value:10.111.206.5,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_PORT,Value:9090,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_ADDR,Value:10.99.30.144,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_PORT,Value:6343,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_HTTP,Value:15672,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_PORT,Value:9000,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT,Value:2200,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP,Value:tcp://10.99.30.144:8043,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT,Value:8004,} &KeyValue{Key:PROMETHEUS_PUBLIC_SERVICE_PORT,Value:9002,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_IPFIX,Value:4739,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP,Value:tcp://10.107.185.128:5672,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_PORT,Value:9000,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_ADDR,Value:10.109.2.52,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_PORT,Value:514,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_ADDR,Value:10.109.91.93,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_ADDR,Value:10.110.117.85,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_PORT,Value:15672,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:GRAFANA_SERVICE_HOST,Value:10.109.208.119,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_HOST,Value:10.110.117.85,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_PORT,Value:4369,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_PROTO,Value:tcp,} &KeyValue{Key:GRAFANA_PORT,Value:tcp://10.109.208.119:9001,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_PORT,Value:2200,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_ADDR,Value:10.99.30.144,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP,Value:tcp://10.105.73.122:80,} &KeyValue{Key:PROMETHEUS_PORT,Value:tcp://10.109.91.93:9002,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_ADDR,Value:10.104.136.1,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_PORT,Value:2100,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT_AWS_LAMBDA,Value:2200,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_3,Value:4369,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP,Value:tcp://10.102.76.76:15672,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP,Value:udp://10.102.19.83:4739,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_2,Value:5672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_4,Value:9090,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_ADDR,Value:10.96.0.1,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_HOST,Value:10.99.30.144,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT_INCOMING,Value:8043,} &KeyValue{Key:GRAFANA_SERVICE_PORT,Value:9001,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_PROTO,Value:tcp,} &KeyValue{Key:DASHBOARD_PORT,Value:tcp://10.104.136.1:9000,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_HOST,Value:10.102.19.83,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_PORT,Value:5672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_PORT,Value:4369,} &KeyValue{Key:DASHBOARD_PUBLIC_SERVICE_PORT,Value:9000,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT,Value:tcp://10.98.195.79:9000,} &KeyValue{Key:PROMETHEUS_PUBLIC_SERVICE_HOST,Value:10.104.141.5,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:METRICS_SERVER_SERVICE_HOST,Value:10.110.5.244,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT_REDIRECT,Value:80,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_PORT,Value:80,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_PORT,Value:9001,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT,Value:tcp://10.111.206.5:443,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_PORT_SFLOW,Value:6343,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT,Value:udp://10.109.2.52:162,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_SERVICE_HOST,Value:10.98.136.14,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP,Value:tcp://10.110.5.244:443,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_ADDR,Value:10.104.141.5,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT,Value:2055,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT,Value:tcp://10.102.76.76:15672,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT_SNMP,Value:162,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP,Value:tcp://10.98.136.14:27017,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_PORT,Value:9001,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_1,Value:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_PORT,Value:15672,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_ADDR,Value:10.105.73.122,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP,Value:udp://10.109.2.52:514,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT,Value:15672,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT_EMI,Value:8004,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_ADDR,Value:10.98.136.14,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_PORT,Value:9090,} &KeyValue{Key:DASHBOARD_PUBLIC_SERVICE_HOST,Value:10.98.195.79,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_ADDR,Value:10.111.206.5,} &KeyValue{Key:KUBERNETES_SERVICE_PORT_HTTPS,Value:443,} &KeyValue{Key:EDGE_RABBITMQ_PORT,Value:tcp://10.107.185.128:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_HOST,Value:10.102.76.76,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_SERVICE_HOST,Value:10.109.91.93,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_ADDR,Value:10.109.2.52,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP,Value:udp://10.110.117.85:6343,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_PORT,Value:5672,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_ADDR,Value:10.98.195.79,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_ADDR,Value:10.104.165.161,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT_SYSLOG,Value:514,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_PORT,Value:6343,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_EXPORTER,Value:9090,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_PORT,Value:9002,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP,Value:tcp://10.99.30.144:2200,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP,Value:tcp://10.104.165.161:9001,} &KeyValue{Key:PROMETHEUS_SERVICE_PORT_HTTP,Value:9002,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_PORT,Value:4739,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_PROTO,Value:tcp,} &KeyValue{Key:METRICS_SERVER_SERVICE_PORT,Value:443,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_SERVICE_PORT,Value:27017,} &KeyValue{Key:EDGE_SFACCTD_PORT,Value:udp://10.110.117.85:6343,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP,Value:tcp://10.107.185.128:4369,} &KeyValue{Key:METRICS_SERVER_PORT,Value:tcp://10.110.5.244:443,} &KeyValue{Key:DASHBOARD_SERVICE_PORT,Value:9000,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP,Value:udp://10.102.19.83:2100,} &KeyValue{Key:KUBERNETES_SERVICE_HOST,Value:10.96.0.1,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP,Value:tcp://10.107.185.128:9090,} &KeyValue{Key:EDGE_EMI_SERVICE_HOST,Value:10.105.73.122,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP,Value:udp://10.109.2.52:162,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_PORT,Value:162,} &KeyValue{Key:GRAFANA_PORT_9001_TCP,Value:tcp://10.109.208.119:9001,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_EPMD,Value:4369,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_ADDR,Value:10.102.76.76,}],Mounts:[&Mount{ContainerPath:/etc/pmacct,HostPath:/var/lib/kubelet/pods/00f37025-0ecc-11e9-ac99-00505691e4af/volumes/kubernetes.io~configmap/sfacctd-volume,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/var/lib/kubelet/pods/00f37025-0ecc-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/default-token-wg9nm,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/00f37025-0ecc-11e9-ac99-00505691e4af/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/00f37025-0ecc-11e9-ac99-00505691e4af/containers/edge-sfacctd/4fa6f54b,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}],Devices:[],Labels:map[string]string{io.kubernetes.container.name: edge-sfacctd,io.kubernetes.pod.name: edge-sfacctd-54bbb77548-brsw6,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 00f37025-0ecc-11e9-ac99-00505691e4af,},Annotations:map[string]string{io.kubernetes.container.hash: 5415deff,io.kubernetes.container.ports: [{"containerPort":6343,"protocol":"UDP"}],io.kubernetes.container.restartCount: 0,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 60,},LogPath:edge-sfacctd/0.log,Stdin:false,StdinOnce:false,Tty:false,Linux:&LinuxContainerConfig{Resources:&LinuxContainerResources{CpuPeriod:100000,CpuQuota:100000,CpuShares:512,MemoryLimitInBytes:805306368,OomScoreAdj:958,CpusetCpus:,CpusetMems:,},SecurityContext:&LinuxContainerSecurityContext{Capabilities:nil,Privileged:false,NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:&Int64Value{Value:0,},RunAsUsername:,ReadonlyRootfs:false,SupplementalGroups:[],ApparmorProfile:,SeccompProfilePath:,NoNewPrivs:false,RunAsGroup:nil,MaskedPaths:[],ReadonlyPaths:[],},},Windows:nil,} ImageRef:sha256:b1d42fde5d0ac7eb628a6e607e8a9ba2af01d657afb3ae653932d755a87f2c01 LogPath:/var/log/pods/00f37025-0ecc-11e9-ac99-00505691e4af/edge-sfacctd/0.log} Status:0xc4208fc200 Container:0xc420866e60 IO:0xc4208ae280 StopCh:0xc4208fe180}"
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.006081134Z" level=debug msg="Start writing stream "stdout" to log file "/var/log/pods/5672db15-0ec3-11e9-ac99-00505691e4af/kube-proxy/0.log""
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.006137480Z" level=debug msg="Start writing stream "stderr" to log file "/var/log/pods/5672db15-0ec3-11e9-ac99-00505691e4af/kube-proxy/0.log""
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.039815800Z" level=debug msg="Loaded container {Metadata:{ID:0f65efaa107732926ac247073b21ea8f012666d2f312d5726ddce96e8b4ea126 Name:kube-proxy_kube-proxy-6wbdx_kube-system_5672db15-0ec3-11e9-ac99-00505691e4af_0 SandboxID:8b2db17fb0c9ecc79e54ae5dbd563fe9969ea11e6ce33af52eb684fa15a83b07 Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:0,},Image:&ImageSpec{Image:sha256:097dd2748285f27a6049caeb404bf5cbeaed8ce5c5fb71f734cf6f125abc9fe6,},Command:[/usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf],Args:[],WorkingDir:,Envs:[&KeyValue{Key:KUBE_DNS_PORT,Value:udp://10.96.0.10:53,} &KeyValue{Key:KUBE_DNS_PORT_53_TCP_ADDR,Value:10.96.0.10,} &KeyValue{Key:TILLER_DEPLOY_SERVICE_PORT,Value:44134,} &KeyValue{Key:KUBERNETES_SERVICE_PORT,Value:443,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:TILLER_DEPLOY_PORT,Value:tcp://10.107.168.180:44134,} &KeyValue{Key:TILLER_DEPLOY_PORT_44134_TCP_ADDR,Value:10.107.168.180,} &KeyValue{Key:KUBERNETES_PORT_443_TCP,Value:tcp://10.96.0.1:443,} &KeyValue{Key:KUBE_DNS_SERVICE_PORT_DNS_TCP,Value:53,} &KeyValue{Key:KUBE_DNS_PORT_53_UDP_ADDR,Value:10.96.0.10,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:KUBE_DNS_SERVICE_HOST,Value:10.96.0.10,} &KeyValue{Key:KUBE_DNS_SERVICE_PORT_DNS,Value:53,} &KeyValue{Key:KUBE_DNS_PORT_53_UDP,Value:udp://10.96.0.10:53,} &KeyValue{Key:KUBE_DNS_PORT_53_UDP_PORT,Value:53,} &KeyValue{Key:KUBE_DNS_PORT_53_TCP_PORT,Value:53,} &KeyValue{Key:TILLER_DEPLOY_PORT_44134_TCP_PROTO,Value:tcp,} &KeyValue{Key:KUBERNETES_SERVICE_PORT_HTTPS,Value:443,} &KeyValue{Key:KUBE_DNS_SERVICE_PORT,Value:53,} &KeyValue{Key:KUBE_DNS_PORT_53_TCP_PROTO,Value:tcp,} &KeyValue{Key:KUBERNETES_PORT,Value:tcp://10.96.0.1:443,} &KeyValue{Key:TILLER_DEPLOY_SERVICE_HOST,Value:10.107.168.180,} &KeyValue{Key:TILLER_DEPLOY_SERVICE_PORT_TILLER,Value:44134,} &KeyValue{Key:TILLER_DEPLOY_PORT_44134_TCP_PORT,Value:44134,} &KeyValue{Key:KUBE_DNS_PORT_53_TCP,Value:tcp://10.96.0.10:53,} &KeyValue{Key:TILLER_DEPLOY_PORT_44134_TCP,Value:tcp://10.107.168.180:44134,} &KeyValue{Key:KUBERNETES_SERVICE_HOST,Value:10.96.0.1,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_ADDR,Value:10.96.0.1,} &KeyValue{Key:KUBE_DNS_PORT_53_UDP_PROTO,Value:udp,}],Mounts:[&Mount{ContainerPath:/var/lib/kube-proxy,HostPath:/var/lib/kubelet/pods/5672db15-0ec3-11e9-ac99-00505691e4af/volumes/kubernetes.io~configmap/kube-proxy,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/run/xtables.lock,HostPath:/run/xtables.lock,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/lib/modules,HostPath:/lib/modules,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/var/lib/kubelet/pods/5672db15-0ec3-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/kube-proxy-token-h5vxk,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/5672db15-0ec3-11e9-ac99-00505691e4af/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/5672db15-0ec3-11e9-ac99-00505691e4af/containers/kube-proxy/d85de9ea,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}],Devices:[],Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-6wbdx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 5672db15-0ec3-11e9-ac99-00505691e4af,},Annotations:map[string]string{io.kubernetes.container.hash: 13176515,io.kubernetes.container.restartCount: 0,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},LogPath:kube-proxy/0.log,Stdin:false,StdinOnce:false,Tty:false,Linux:&LinuxContainerConfig{Resources:&LinuxContainerResources{CpuPeriod:0,CpuQuota:0,CpuShares:2,MemoryLimitInBytes:0,OomScoreAdj:1000,CpusetCpus:,CpusetMems:,},SecurityContext:&LinuxContainerSecurityContext{Capabilities:nil,Privileged:true,NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:&Int64Value{Value:0,},RunAsUsername:,ReadonlyRootfs:false,SupplementalGroups:[],ApparmorProfile:,SeccompProfilePath:,NoNewPrivs:false,RunAsGroup:nil,MaskedPaths:[],ReadonlyPaths:[],},},Windows:nil,} ImageRef:sha256:097dd2748285f27a6049caeb404bf5cbeaed8ce5c5fb71f734cf6f125abc9fe6 LogPath:/var/log/pods/5672db15-0ec3-11e9-ac99-00505691e4af/kube-proxy/0.log} Status:0xc420930180 Container:0xc420866e80 IO:0xc42034f880 StopCh:0xc420764460}"
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.049199551Z" level=debug msg="Loaded container {Metadata:{ID:290fb9bbf983bee091c3bf50676df4648319e347c38c9dcc533137e459be35b9 Name:edge-inventory_edge-inventory-669b6ff887-qtm5z_production_4f103bac-0ec4-11e9-ac99-00505691e4af_0 SandboxID:fab4e9c2bcadf1a652fe840cdc4a87c46d08e856ce3568bb8ed9e7f2c18ed350 Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:edge-inventory,Attempt:0,},Image:&ImageSpec{Image:sha256:9401a94b4fa06363d66435934ccb0ee0d8854eaf5660e8008909e16521316b87,},Command:[],Args:[],WorkingDir:,Envs:[&KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP,Value:udp://10.102.19.83:2055,} &KeyValue{Key:KUBERNETES_PORT_443_TCP,Value:tcp://10.96.0.1:443,} &KeyValue{Key:DASHBOARD_PUBLIC_SERVICE_HOST,Value:10.98.195.79,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP,Value:udp://10.110.117.85:6343,} &KeyValue{Key:PROMETHEUS_SERVICE_HOST,Value:10.109.91.93,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT_SNMP,Value:162,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT,Value:udp://10.109.2.52:162,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_PORT,Value:9002,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP,Value:tcp://10.102.76.76:5672,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_ADDR,Value:10.96.0.1,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_ADDR,Value:10.98.136.14,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:GRAFANA_PORT_9001_TCP,Value:tcp://10.109.208.119:9001,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_PORT,Value:9090,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_PORT,Value:9000,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_PORT,Value:4739,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_PROTO,Value:tcp,} &KeyValue{Key:KUBERNETES_SERVICE_HOST,Value:10.96.0.1,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP,Value:tcp://10.111.206.5:443,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_PORT,Value:9002,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP,Value:tcp://10.104.141.5:9002,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT_REDIRECT,Value:80,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_PORT,Value:9001,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_ADDR,Value:10.111.206.5,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_NETFLOW2,Value:2100,} &KeyValue{Key:EDGE_NFACCTD_PORT,Value:udp://10.102.19.83:2055,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_PORT,Value:9000,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT,Value:tcp://10.111.206.5:443,} &KeyValue{Key:DASHBOARD_PUBLIC_SERVICE_PORT,Value:9000,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_PORT,Value:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP,Value:tcp://10.102.76.76:4369,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP,Value:tcp://10.99.30.144:2200,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_PORT,Value:2200,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_ADDR,Value:10.104.141.5,} &KeyValue{Key:EDGE_SFACCTD_PORT,Value:udp://10.110.117.85:6343,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT,Value:162,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_HOST,Value:10.107.185.128,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_2,Value:5672,} &KeyValue{Key:KUBERNETES_PORT,Value:tcp://10.96.0.1:443,} &KeyValue{Key:GRAFANA_PUBLIC_SERVICE_PORT,Value:9001,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_HOST,Value:10.102.76.76,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT,Value:tcp://10.99.30.144:2200,} &KeyValue{Key:METRICS_SERVER_SERVICE_PORT,Value:443,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_NETFLOW1,Value:2055,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_PROTO,Value:udp,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_PORT,Value:4369,} &KeyValue{Key:PROMETHEUS_PORT,Value:tcp://10.109.91.93:9002,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_PORT,Value:514,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_AMQP,Value:5672,} &KeyValue{Key:PROMETHEUS_ADAPTER_PORT_443_TCP_PORT,Value:443,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_ADDR,Value:10.109.91.93,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_PROTO,Value:tcp,} &KeyValue{Key:PROMETHEUS_PUBLIC_SERVICE_HOST,Value:10.104.141.5,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_PORT,Value:80,} &KeyValue{Key:METRICS_SERVER_PORT,Value:tcp://10.110.5.244:443,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_EXPORTER,Value:9090,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT,Value:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_3,Value:4369,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP,Value:udp://10.109.2.52:514,} &KeyValue{Key:KUBERNETES_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP,Value:tcp://10.105.73.122:80,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_ADDR,Value:10.104.165.161,} &KeyValue{Key:EDGE_RABBITMQ_PORT,Value:tcp://10.107.185.128:15672,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP,Value:udp://10.102.19.83:2100,} &KeyValue{Key:KUBERNETES_SERVICE_PORT,Value:443,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP,Value:tcp://10.105.73.122:8004,} &KeyValue{Key:GRAFANA_SERVICE_HOST,Value:10.109.208.119,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_HTTP,Value:15672,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP,Value:udp://10.109.2.52:162,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_HOST,Value:10.99.30.144,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_PROTO,Value:tcp,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_ADDR,Value:10.110.5.244,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_PORT,Value:9001,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP,Value:tcp://10.98.195.79:9000,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP,Value:tcp://10.102.76.76:15672,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_PORT,Value:8043,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT_9002_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_EMI_SERVICE_HOST,Value:10.105.73.122,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_PORT,Value:5672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_5672_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_PORT,Value:6343,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT,Value:tcp://10.98.136.14:27017,} &KeyValue{Key:GRAFANA_PUBLIC_SERVICE_HOST,Value:10.104.165.161,} &KeyValue{Key:GRAFANA_PORT_9001_TCP_ADDR,Value:10.109.208.119,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP,Value:tcp://10.107.185.128:5672,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP,Value:tcp://10.107.185.128:4369,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_4,Value:9090,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_SERVICE_PORT_PORT_1,Value:15672,} &KeyValue{Key:PROMETHEUS_SERVICE_PORT,Value:9002,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_PORT_SYSLOG,Value:514,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_SERVICE_HOST,Value:10.98.136.14,} &KeyValue{Key:DASHBOARD_PORT,Value:tcp://10.104.136.1:9000,} &KeyValue{Key:EDGE_RABBITMQ_PORT_5672_TCP_PORT,Value:5672,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP,Value:tcp://10.109.91.93:9002,} &KeyValue{Key:GRAFANA_PUBLIC_PORT,Value:tcp://10.104.165.161:9001,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP,Value:tcp://10.104.165.161:9001,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP,Value:tcp://10.102.76.76:9090,} &KeyValue{Key:GRAFANA_SERVICE_PORT,Value:9001,} &KeyValue{Key:PROMETHEUS_PORT_9002_TCP_PROTO,Value:tcp,} &KeyValue{Key:METRICS_SERVER_SERVICE_HOST,Value:10.110.5.244,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP,Value:tcp://10.110.5.244:443,} &KeyValue{Key:DASHBOARD_SERVICE_HOST,Value:10.104.136.1,} &KeyValue{Key:DASHBOARD_SERVICE_PORT,Value:9000,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP_ADDR,Value:10.104.136.1,} &KeyValue{Key:GRAFANA_PORT,Value:tcp://10.109.208.119:9001,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP,Value:tcp://10.107.185.128:9090,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:PROMETHEUS_PUBLIC_SERVICE_PORT,Value:9002,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_9090_TCP_PORT,Value:9090,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_PORT,Value:6343,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT_EMI,Value:8004,} &KeyValue{Key:EDGE_RABBITMQ_PORT_9090_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_PORT,Value:162,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT_IPFIX,Value:4739,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP,Value:udp://10.102.19.83:4739,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT_AWS_LAMBDA,Value:2200,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_SFACCTD_PORT_6343_UDP_ADDR,Value:10.110.117.85,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_PORT,Value:2055,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_ADDR,Value:10.102.19.83,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP,Value:tcp://10.99.30.144:8043,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP,Value:tcp://10.107.185.128:15672,} &KeyValue{Key:EDGE_NFACCTD_SERVICE_HOST,Value:10.102.19.83,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT_INCOMING,Value:8043,} &KeyValue{Key:DASHBOARD_PORT_9000_TCP,Value:tcp://10.104.136.1:9000,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_PORT,Value:8004,} &KeyValue{Key:PROMETHEUS_ADAPTER_SERVICE_PORT,Value:443,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_2200_TCP_ADDR,Value:10.99.30.144,} &KeyValue{Key:PROMETHEUS_PUBLIC_PORT,Value:tcp://10.104.141.5:9002,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_SERVICE_PORT,Value:27017,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP,Value:tcp://10.98.136.14:27017,} &KeyValue{Key:EDGE_RABBITMQ_PORT_15672_TCP_PORT,Value:15672,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_HOST,Value:10.110.117.85,} &KeyValue{Key:PROMETHEUS_SERVICE_PORT_HTTP,Value:9002,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_514_UDP_ADDR,Value:10.109.2.52,} &KeyValue{Key:EDGE_NFACCTD_PORT_4739_UDP_PROTO,Value:udp,} &KeyValue{Key:EDGE_INBOUND_TCP_SERVICE_PORT,Value:2200,} &KeyValue{Key:KUBERNETES_SERVICE_PORT_HTTPS,Value:443,} &KeyValue{Key:EDGE_RABBITMQ_PORT_4369_TCP_ADDR,Value:10.107.185.128,} &KeyValue{Key:EDGE_SFACCTD_SERVICE_PORT_SFLOW,Value:6343,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_PROTO,Value:udp,} &KeyValue{Key:METRICS_SERVER_PORT_443_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT,Value:15672,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT_9000_TCP_ADDR,Value:10.98.195.79,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_15672_TCP_ADDR,Value:10.102.76.76,} &KeyValue{Key:EDGE_EMI_SERVICE_PORT,Value:8004,} &KeyValue{Key:GRAFANA_PUBLIC_PORT_9001_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_RABBITMQ_SERVICE_PORT_EPMD,Value:4369,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_PROTO,Value:tcp,} &KeyValue{Key:EDGE_MONGODB_CLIENT_PUBLIC_PORT_27017_TCP_PORT,Value:27017,} &KeyValue{Key:EDGE_EMI_PORT,Value:tcp://10.105.73.122:8004,} &KeyValue{Key:EDGE_EMI_PORT_8004_TCP_ADDR,Value:10.105.73.122,} &KeyValue{Key:EDGE_EMI_PORT_80_TCP_ADDR,Value:10.105.73.122,} &KeyValue{Key:EDGE_NFACCTD_PORT_2055_UDP_PORT,Value:2055,} &KeyValue{Key:EDGE_NFACCTD_PORT_2100_UDP_PORT,Value:2100,} &KeyValue{Key:EDGE_INBOUND_TCP_PORT_8043_TCP_ADDR,Value:10.99.30.144,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT,Value:tcp://10.102.76.76:15672,} &KeyValue{Key:EDGE_RABBITMQ_PUBLIC_PORT_4369_TCP_PORT,Value:4369,} &KeyValue{Key:EDGE_INBOUND_UDP_SERVICE_HOST,Value:10.109.2.52,} &KeyValue{Key:EDGE_INBOUND_UDP_PORT_162_UDP_ADDR,Value:10.109.2.52,} &KeyValue{Key:GRAFANA_SERVICE_PORT_SERVICE,Value:9001,} &KeyValue{Key:PROMETHEUS_ADAPTER_SERVICE_HOST,Value:10.111.206.5,} &KeyValue{Key:DASHBOARD_PUBLIC_PORT,Value:tcp://10.98.195.79:9000,}],Mounts:[&Mount{ContainerPath:/etc/edge/mongodb,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/mongodb,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/edge/proxy,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/proxy,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/edge/rabbitmq,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/rabbitmq,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/var/run/secrets/kubernetes.io/serviceaccount,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/volumes/kubernetes.io~secret/default-token-wg9nm,Readonly:true,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/etc/hosts,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/etc-hosts,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,} &Mount{ContainerPath:/dev/termination-log,HostPath:/var/lib/kubelet/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/containers/edge-inventory/302e1c2c,Readonly:false,SelinuxRelabel:false,Propagation:PROPAGATION_PRIVATE,}],Devices:[],Labels:map[string]string{io.kubernetes.container.name: edge-inventory,io.kubernetes.pod.name: edge-inventory-669b6ff887-qtm5z,io.kubernetes.pod.namespace: production,io.kubernetes.pod.uid: 4f103bac-0ec4-11e9-ac99-00505691e4af,},Annotations:map[string]string{io.kubernetes.container.hash: 57fad244,io.kubernetes.container.ports: [{"hostPort":8084,"containerPort":8084,"protocol":"TCP"}],io.kubernetes.container.restartCount: 0,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 60,},LogPath:edge-inventory/0.log,Stdin:false,StdinOnce:false,Tty:false,Linux:&LinuxContainerConfig{Resources:&LinuxContainerResources{CpuPeriod:100000,CpuQuota:100000,CpuShares:512,MemoryLimitInBytes:2147483648,OomScoreAdj:938,CpusetCpus:,CpusetMems:,},SecurityContext:&LinuxContainerSecurityContext{Capabilities:nil,Privileged:false,NamespaceOptions:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,},SelinuxOptions:nil,RunAsUser:&Int64Value{Value:0,},RunAsUsername:,ReadonlyRootfs:false,SupplementalGroups:[],ApparmorProfile:,SeccompProfilePath:,NoNewPrivs:false,RunAsGroup:nil,MaskedPaths:[],ReadonlyPaths:[],},},Windows:nil,} ImageRef:sha256:9401a94b4fa06363d66435934ccb0ee0d8854eaf5660e8008909e16521316b87 LogPath:/var/log/pods/4f103bac-0ec4-11e9-ac99-00505691e4af/edge-inventory/0.log} Status:0xc420930200 Container:0xc420866ec0 IO:<nil> StopCh:0xc42032e6a0}"
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.079751837Z" level=debug msg="Start writing stream "stdout" to log file "/var/log/pods/d6448f2a-e152-11e8-852a-00505691e4af/weave/0.log""
Jan 03 19:54:43 qaedge1fce1-2 containerd[242538]: time="2019-01-03T19:54:43.079805166Z" level=debug msg="Start writing stream "stderr" to log file "/var/log/pods/d6448f2a-e152-11e8-852a-00505691e4af/weave/0.log""
sudo ctr pprof goroutines
goroutine 3628 [running]:
runtime/pprof.writeGoroutineStacks(0x556616cf4200, 0xc4205381c0, 0x55661588aaf5, 0xc420695a70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/pprof/pprof.go:650 +0xa9
runtime/pprof.writeGoroutine(0x556616cf4200, 0xc4205381c0, 0x2, 0xc42006b400, 0x556616cf3fe0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/pprof/pprof.go:639 +0x46
runtime/pprof.(*Profile).WriteTo(0x556617438dc0, 0x556616cf4200, 0xc4205381c0, 0x2, 0xc4205381c0, 0x55661654e2f8)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/pprof/pprof.go:310 +0x3e6
net/http/pprof.handler.ServeHTTP(0xc42089be41, 0x9, 0x556616d0bfc0, 0xc4205381c0, 0xc4200e2400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/pprof/pprof.go:243 +0x20f
net/http/pprof.Index(0x556616d0bfc0, 0xc4205381c0, 0xc4200e2400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/pprof/pprof.go:254 +0x1d3
net/http.HandlerFunc.ServeHTTP(0x556616cea490, 0x556616d0bfc0, 0xc4205381c0, 0xc4200e2400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:1947 +0x46
net/http.(*ServeMux).ServeHTTP(0xc4205be930, 0x556616d0bfc0, 0xc4205381c0, 0xc4200e2400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:2340 +0x132
net/http.serverHandler.ServeHTTP(0xc420390000, 0x556616d0bfc0, 0xc4205381c0, 0xc4200e2400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:2697 +0xbe
net/http.(*conn).serve(0xc4202a1f40, 0x556616d0d2c0, 0xc4202c61c0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:1830 +0x653
created by net/http.(*Server).Serve
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:2798 +0x27d

goroutine 1 [chan receive, 26 minutes]:
github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc420126160, 0xc420126160, 0xc42053dd07)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:174 +0xa01
github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x556616adbf00, 0x556616ce7a90, 0xc420126160, 0xc420590c00, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:502 +0xca
github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc42039e540, 0xc42003a1b0, 0x1, 0x1, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:268 +0x60e
main.main()
    github.com/containerd/containerd/cmd/containerd/main.go:33 +0x51

goroutine 5 [syscall, 26 minutes]:
os/signal.signal_recv(0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/sigqueue.go:139 +0xa8
os/signal.loop()
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/signal/signal_unix.go:22 +0x24
created by os/signal.init.0
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/signal/signal_unix.go:28 +0x43

goroutine 6 [chan receive, 1 minutes]:
github.com/containerd/containerd/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x556617492e40)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:879 +0x8d
created by github.com/containerd/containerd/vendor/github.com/golang/glog.init.0
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/golang/glog/glog.go:410 +0x205

goroutine 13 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5cf00, 0x72, 0xc4201f0b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008a198, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008a198, 0xc4201fb000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008a180, 0xc4201fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008a180, 0xc4201fb000, 0x1000, 0x1000, 0xc4201f0c68, 0x5566158d1f10, 0xc4204f6480)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e088, 0xc4201fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420386660, 0xc42034ed20, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420386660, 0xc42034ed20, 0xa, 0xa, 0xa, 0xc4200a0d01, 0x0, 0xc4203724e0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420386660, 0xc42034ed20, 0xa, 0xa, 0xc4201f0f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc42034ed20, 0xa, 0xa, 0x556616cf24a0, 0xc420386660, 0xc4201f0f48, 0x2, 0x2, 0xc4201f0fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc42034ed00, 0x556616d0d300, 0xc42003e040, 0x0, 0x5566158d0002, 0xc420934000, 0x270, 0x270, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4205e4600, 0xc4203866c0, 0xc4201e4900, 0xc4205e45a0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 74 [select, 26 minutes]:
github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1(0xc420590cc0, 0xc420590c60, 0x556616d0d300, 0xc42003e018, 0xc4200a22a0)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:46 +0xea
created by github.com/containerd/containerd/cmd/containerd/command.handleSignals
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main_unix.go:43 +0x8b

goroutine 75 [select, 26 minutes, locked to thread]:
runtime.gopark(0x556616cea900, 0x0, 0x556616546231, 0x6, 0x18, 0x1)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/proc.go:291 +0x120
runtime.selectgo(0xc4204f3750, 0xc4200a2360)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/select.go:392 +0xe56
runtime.ensureSigM.func1()
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/signal_unix.go:549 +0x1f6
runtime.goexit()
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1

goroutine 76 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Broadcaster).run(0xc42038aa00)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:117 +0x3c4
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewBroadcaster
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b1

goroutine 20 [select]:
github.com/containerd/containerd/gc/scheduler.(*gcScheduler).run(0xc420386540, 0x556616d0d300, 0xc42003e018)
    /home/travis/gopath/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:268 +0x1e9
created by github.com/containerd/containerd/gc/scheduler.init.0.func1
    /home/travis/gopath/src/github.com/containerd/containerd/gc/scheduler/scheduler.go:132 +0x490

goroutine 21 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4203866c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 22 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420386840)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 79 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5ce30, 0x72, 0xc420075b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008a598, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008a598, 0xc4202fe000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008a580, 0xc4202fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008a580, 0xc4202fe000, 0x1000, 0x1000, 0xc420075c68, 0x5566158d1f10, 0xc4204f6780)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e148, 0xc4202fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc4203867e0, 0xc42034f620, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc4203867e0, 0xc42034f620, 0xa, 0xa, 0xa, 0xc420513001, 0x0, 0xc420373d40)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc4203867e0, 0xc42034f620, 0xa, 0xa, 0xc420075f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc42034f620, 0xa, 0xa, 0x556616cf24a0, 0xc4203867e0, 0xc420075f48, 0x2, 0x2, 0xc420075fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc42034f600, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc4200ec000, 0x270, 0x270, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200a2de0, 0xc420386840, 0xc42062ceb0, 0xc4200a2d80)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 15 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4205901e0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 51 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5cd60, 0x72, 0xc42007ab80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4298, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4298, 0xc4202ff000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4280, 0xc4202ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4280, 0xc4202ff000, 0x1000, 0x1000, 0xc42007ac68, 0x5566158d1f10, 0xc42010ea80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c0a8, 0xc4202ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590180, 0xc420570620, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590180, 0xc420570620, 0xa, 0xa, 0xa, 0xc4200a0c01, 0x0, 0xc420372000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590180, 0xc420570620, 0xa, 0xa, 0xc42007af48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420570620, 0xa, 0xa, 0x556616cf24a0, 0xc420590180, 0xc42007af48, 0x2, 0x2, 0xc42007afa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420570600, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc420260000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc420372060, 0xc4205901e0, 0xc42056c000, 0xc420372000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 16 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590360)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 23 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5cc90, 0x72, 0xc4201eeb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4698, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4698, 0xc42029e000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4680, 0xc42029e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4680, 0xc42029e000, 0x1000, 0x1000, 0xc4201eec68, 0x5566158d1f10, 0xc42010ed80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c158, 0xc42029e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590300, 0xc420571760, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590300, 0xc420571760, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc4205e4000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590300, 0xc420571760, 0xa, 0xa, 0xc4201eef48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420571760, 0xa, 0xa, 0x556616cf24a0, 0xc420590300, 0xc4201eef48, 0x2, 0x2, 0xc4201eefa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420571740, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4205e4060, 0xc420590360, 0xc42003a010, 0xc4205e4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 82 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4205905a0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 24 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5cbc0, 0x72, 0xc4201f2b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4a98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4a98, 0xc420369000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4a80, 0xc420369000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4a80, 0xc420369000, 0x1000, 0x1000, 0xc4201f2c68, 0x5566158d1f10, 0xc42010f080)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c200, 0xc420369000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590540, 0xc4203340a0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590540, 0xc4203340a0, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc42030baa0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590540, 0xc4203340a0, 0xa, 0xa, 0xc4201f2f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4203340a0, 0xa, 0xa, 0x556616cf24a0, 0xc420590540, 0xc4201f2f48, 0x2, 0x2, 0xc4201f2fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420334080, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4205e4120, 0xc4205905a0, 0xc42003a380, 0xc4205e40c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 83 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4205907e0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 25 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5caf0, 0x72, 0xc4203fcb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4e98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4e98, 0xc420318000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4e80, 0xc420318000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4e80, 0xc420318000, 0x1000, 0x1000, 0xc4203fcc68, 0x5566158d1f10, 0xc42010f380)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c2a8, 0xc420318000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590780, 0xc4203349e0, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590780, 0xc4203349e0, 0xa, 0xa, 0xa, 0xc420513080, 0x0, 0xc4205e4180)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590780, 0xc4203349e0, 0xa, 0xa, 0xc4203fcf48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4203349e0, 0xa, 0xa, 0x556616cf24a0, 0xc420590780, 0xc4203fcf48, 0x2, 0x2, 0xc4203fcfa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4203349c0, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42069e500, 0x273, 0x273, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4205e41e0, 0xc4205907e0, 0xc42003aa30, 0xc4205e4180)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 84 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590a20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 37 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5ca20, 0x72, 0xc4203f8b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5298, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5298, 0xc42031c000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5280, 0xc42031c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5280, 0xc42031c000, 0x1000, 0x1000, 0x1, 0x0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c2d8, 0xc42031c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc4205909c0, 0xc420334ba0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc4205909c0, 0xc420334ba0, 0xa, 0xa, 0xa, 0xa695c57500000000, 0xc420064e70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc4205909c0, 0xc420334ba0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420334ba0, 0xa, 0xa, 0x556616cf24a0, 0xc4205909c0, 0x0, 0x5566158d1a10, 0x8, 0xc420064e20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420334b80, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42031e060, 0xc420590a20, 0xc420292000, 0xc42031e000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 85 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590ba0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 52 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c950, 0x72, 0xc420076b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5718, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5718, 0xc420325000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5700, 0xc420325000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5700, 0xc420325000, 0x1000, 0x1000, 0xc420076c68, 0x5566158d1f10, 0xc42010f980)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c380, 0xc420325000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590b40, 0xc4203354a0, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590b40, 0xc4203354a0, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc4203720c0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590b40, 0xc4203354a0, 0xa, 0xa, 0xc420076f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4203354a0, 0xa, 0xa, 0x556616cf24a0, 0xc420590b40, 0xc420076f48, 0x2, 0x2, 0xc420076fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420335480, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc420372120, 0xc420590ba0, 0xc42056c010, 0xc4203720c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 86 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590f00)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 38 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c880, 0x72, 0xc4203fab80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5b98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5b98, 0xc4202de000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5b80, 0xc4202de000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5b80, 0xc4202de000, 0x1000, 0x1000, 0xc4203fac68, 0x5566158d1f10, 0xc42010fc80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c428, 0xc4202de000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590ea0, 0xc420335da0, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590ea0, 0xc420335da0, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc42031e0c0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590ea0, 0xc420335da0, 0xa, 0xa, 0xc4203faf48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420335da0, 0xa, 0xa, 0x556616cf24a0, 0xc420590ea0, 0xc4203faf48, 0x2, 0x2, 0xc4203fafa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420335d80, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42031e120, 0xc420590f00, 0xc420292010, 0xc42031e0c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 87 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4205913e0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 53 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c7b0, 0x72, 0xc42052cb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008a118, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008a118, 0xc420621000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008a100, 0xc420621000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008a100, 0xc420621000, 0x1000, 0x1000, 0x1, 0x0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c4d0, 0xc420621000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420591380, 0xc4202ec6a0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420591380, 0xc4202ec6a0, 0xa, 0xa, 0xa, 0x796d01cf00000000, 0xc4204eee70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420591380, 0xc4202ec6a0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202ec6a0, 0xa, 0xa, 0x556616cf24a0, 0xc420591380, 0x0, 0x5566158d1a10, 0x8, 0xc4204eee20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202ec680, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4203721e0, 0xc4205913e0, 0xc42056c030, 0xc420372180)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 88 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420591560)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 54 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c6e0, 0x72, 0xc4203f6b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008a618, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008a618, 0xc420627000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008a600, 0xc420627000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008a600, 0xc420627000, 0x1000, 0x1000, 0xc4203f6c68, 0x5566158d1f10, 0xc4203a3200)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c500, 0xc420627000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420591500, 0xc4202ec860, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420591500, 0xc4202ec860, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc420372240)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420591500, 0xc4202ec860, 0xa, 0xa, 0xc4203f6f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202ec860, 0xa, 0xa, 0x556616cf24a0, 0xc420591500, 0xc4203f6f48, 0x2, 0x2, 0xc4203f6fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202ec840, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4203722a0, 0xc420591560, 0xc42056c040, 0xc420372240)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 89 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc4205916e0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 55 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c610, 0x72, 0xc420528b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008aa18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008aa18, 0xc420250000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008aa00, 0xc420250000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008aa00, 0xc420250000, 0x1000, 0x1000, 0x1, 0x0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c5a8, 0xc420250000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420591680, 0xc4202ed160, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420591680, 0xc4202ed160, 0xa, 0xa, 0xa, 0x796d01cf00000000, 0xc4204eee70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420591680, 0xc4202ed160, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202ed160, 0xa, 0xa, 0x556616cf24a0, 0xc420591680, 0x0, 0x5566158d1a10, 0x8, 0xc4204eee20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202ed140, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4203723c0, 0xc4205916e0, 0xc42056c060, 0xc420372300)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 90 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420591860)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 39 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c540, 0x72, 0xc420666b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008ae18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008ae18, 0xc42063b000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008ae00, 0xc42063b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008ae00, 0xc42063b000, 0x1000, 0x1000, 0xc420666c68, 0x5566158d1f10, 0xc4203a3800)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c650, 0xc42063b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420591800, 0xc4202eda60, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420591800, 0xc4202eda60, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc420262f00)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420591800, 0xc4202eda60, 0xa, 0xa, 0xc420666f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202eda60, 0xa, 0xa, 0x556616cf24a0, 0xc420591800, 0xc420666f48, 0x2, 0x2, 0xc420666fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202eda40, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42031e1e0, 0xc420591860, 0xc420292020, 0xc42031e180)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 26 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387080)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 91 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c470, 0x72, 0xc420526b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4318, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4318, 0xc420319000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4300, 0xc420319000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4300, 0xc420319000, 0x1000, 0x1000, 0xc400000001, 0xc4205be900, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e080, 0xc420319000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387020, 0xc4205708a0, 0xa, 0xa, 0x0, 0xc42052f6e0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387020, 0xc4205708a0, 0xa, 0xa, 0xa, 0xe2bf08a00000000, 0xc4204f2670, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387020, 0xc4205708a0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4205708a0, 0xa, 0xa, 0x556616cf24a0, 0xc420387020, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420570880, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f27c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a060, 0xc420387080, 0xc4201e4000, 0xc42030a000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 27 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387200)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 40 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c3a0, 0x72, 0xc420664b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4798, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4798, 0xc4202df000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4780, 0xc4202df000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4780, 0xc4202df000, 0x1000, 0x1000, 0xc420664c68, 0x5566158d1f10, 0xc42010f200)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e0c8, 0xc4202df000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc4203871a0, 0xc420570be0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc4203871a0, 0xc420570be0, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc4200a3260)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc4203871a0, 0xc420570be0, 0xa, 0xa, 0xc420664f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420570be0, 0xa, 0xa, 0x556616cf24a0, 0xc4203871a0, 0xc420664f48, 0x2, 0x2, 0xc420664fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420570bc0, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c6060, 0xc420387200, 0xc42056c240, 0xc4200c6000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 28 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387380)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 92 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c2d0, 0x72, 0xc42052ab80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4c18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4c18, 0xc42020c000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4c00, 0xc42020c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4c00, 0xc42020c000, 0x1000, 0x1000, 0xc42052ac68, 0x5566158d1f10, 0xc42010f800)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e188, 0xc42020c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387320, 0xc420571ca0, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387320, 0xc420571ca0, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc42030a0c0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387320, 0xc420571ca0, 0xa, 0xa, 0xc42052af48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420571ca0, 0xa, 0xa, 0x556616cf24a0, 0xc420387320, 0xc42052af48, 0x2, 0x2, 0xc42052afa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420571c80, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a120, 0xc420387380, 0xc4201e4010, 0xc42030a0c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 29 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387500)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 41 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c200, 0x72, 0xc4203fdb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5098, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5098, 0xc42061f000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5080, 0xc42061f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5080, 0xc42061f000, 0x1000, 0x1000, 0xc400000001, 0xc4205be8a0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e230, 0xc42061f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc4203874a0, 0xc420300820, 0xa, 0xa, 0x0, 0xc4205f9f20, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc4203874a0, 0xc420300820, 0xa, 0xa, 0xa, 0xa983fc7000000000, 0xc4204f1e70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc4203874a0, 0xc420300820, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420300820, 0xa, 0xa, 0x556616cf24a0, 0xc4203874a0, 0x0, 0x5566158d1a10, 0x8, 0xc4204f1e20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420300800, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f1fc0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c6120, 0xc420387500, 0xc42056c260, 0xc4200c60c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 30 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387680)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 93 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c130, 0x72, 0xc420660b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5518, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5518, 0xc4201e0000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5500, 0xc4201e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5500, 0xc4201e0000, 0x1000, 0x1000, 0xc400000001, 0xc4205be900, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e2d8, 0xc4201e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387620, 0xc420301120, 0xa, 0xa, 0x0, 0xc42052f6e0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387620, 0xc420301120, 0xa, 0xa, 0xa, 0xe2bf08a00000000, 0xc4204f2670, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387620, 0xc420301120, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420301120, 0xa, 0xa, 0x556616cf24a0, 0xc420387620, 0x0, 0x5566158d1a10, 0x8, 0xc4204f2620)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420301100, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f27c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a360, 0xc420387680, 0xc4201e4020, 0xc42030a180)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 31 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387800)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 42 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796c5c060, 0x72, 0xc42064eb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5a18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5a18, 0xc4202ae000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5a00, 0xc4202ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5a00, 0xc4202ae000, 0x1000, 0x1000, 0xc400000001, 0xc4205be8a0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e308, 0xc4202ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc4203877a0, 0xc4203012e0, 0xa, 0xa, 0x0, 0xc4205f9f20, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc4203877a0, 0xc4203012e0, 0xa, 0xa, 0xa, 0xa983fc7000000000, 0xc4204f1e70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc4203877a0, 0xc4203012e0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4203012e0, 0xa, 0xa, 0x556616cf24a0, 0xc4203877a0, 0x0, 0x5566158d1a10, 0x8, 0xc4204f1e20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4203012c0, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f1fc0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c61e0, 0xc420387800, 0xc42056c290, 0xc4200c6180)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 32 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387980)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 94 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a6e8, 0x72, 0xc420662b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a5e98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a5e98, 0xc420356000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a5e80, 0xc420356000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a5e80, 0xc420356000, 0x1000, 0x1000, 0xc400000001, 0xc4205be900, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e338, 0xc420356000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387920, 0xc4203014a0, 0xa, 0xa, 0x0, 0xc42052f6e0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387920, 0xc4203014a0, 0xa, 0xa, 0xa, 0xe2bf08a00000000, 0xc4204f2670, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387920, 0xc4203014a0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4203014a0, 0xa, 0xa, 0x556616cf24a0, 0xc420387920, 0x0, 0x5566158d1a10, 0x8, 0xc4204f2620)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420301480, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f27c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a420, 0xc420387980, 0xc4201e4030, 0xc42030a3c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 33 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387b00)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 43 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a618, 0x72, 0xc420650b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008a518, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008a518, 0xc420087000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008a500, 0xc420087000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008a500, 0xc420087000, 0x1000, 0x1000, 0xc400000001, 0xc4205be8a0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e3e0, 0xc420087000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387aa0, 0xc420301da0, 0xa, 0xa, 0x0, 0xc4205f9f20, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387aa0, 0xc420301da0, 0xa, 0xa, 0xa, 0xa983fc7000000000, 0xc4204f1e70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387aa0, 0xc420301da0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420301da0, 0xa, 0xa, 0x556616cf24a0, 0xc420387aa0, 0x0, 0x5566158d1a10, 0x8, 0xc4204f1e20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420301d80, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f1fc0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c62a0, 0xc420387b00, 0xc42056c2b0, 0xc4200c6240)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 98 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387c80)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 95 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a548, 0x72, 0xc42064ab80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008aa98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008aa98, 0xc420305000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008aa80, 0xc420305000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008aa80, 0xc420305000, 0x1000, 0x1000, 0xc42064ac68, 0x5566158d1f10, 0xc4203a3380)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e410, 0xc420305000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387c20, 0xc420301f60, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387c20, 0xc420301f60, 0xa, 0xa, 0xa, 0xc4200a0e80, 0x0, 0xc420372a80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387c20, 0xc420301f60, 0xa, 0xa, 0xc42064af48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc420301f60, 0xa, 0xa, 0x556616cf24a0, 0xc420387c20, 0xc42064af48, 0x2, 0x2, 0xc42064afa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc420301f40, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc42002a000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a4e0, 0xc420387c80, 0xc4201e4040, 0xc42030a480)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 99 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387e00)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 44 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a478, 0x72, 0xc420360b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008af18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008af18, 0xc42025a000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008af00, 0xc42025a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008af00, 0xc42025a000, 0x1000, 0x1000, 0xc420360c68, 0x5566158d1f10, 0xc4203a3e00)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e4c0, 0xc42025a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387da0, 0xc42003d5a0, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387da0, 0xc42003d5a0, 0xa, 0xa, 0xa, 0xc4200a0c80, 0x0, 0xc4200c6300)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387da0, 0xc42003d5a0, 0xa, 0xa, 0xc420360f48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc42003d5a0, 0xa, 0xa, 0x556616cf24a0, 0xc420387da0, 0xc420360f48, 0x2, 0x2, 0xc420360fa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc42003d580, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc420260000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c6360, 0xc420387e00, 0xc42056c2e0, 0xc4200c6300)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 100 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420387f80)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 45 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a3a8, 0x72, 0xc420362b80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008b318, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008b318, 0xc420327000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008b300, 0xc420327000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008b300, 0xc420327000, 0x1000, 0x1000, 0xc400000001, 0xc4205be8a0, 0x556616a7ee20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e5e8, 0xc420327000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420387f20, 0xc42003dea0, 0xa, 0xa, 0x0, 0xc4205f9f20, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420387f20, 0xc42003dea0, 0xa, 0xa, 0xa, 0xa983fc7000000000, 0xc4204f1e70, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420387f20, 0xc42003dea0, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc42003dea0, 0xa, 0xa, 0x556616cf24a0, 0xc420387f20, 0x0, 0x5566158d1a10, 0x8, 0xc4204f1e20)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc42003de80, 0x556616d0d300, 0xc42003e040, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40004, 0xc4204f1fc0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc4200c6420, 0xc420387f80, 0xc42056c300, 0xc4200c63c0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 101 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590720)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 96 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a2d8, 0x72, 0xc42064cb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008b718, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008b718, 0xc42032b000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008b700, 0xc42032b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008b700, 0xc42032b000, 0x1000, 0x1000, 0xc42064cc68, 0x5566158d1f10, 0xc420310900)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e618, 0xc42032b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590420, 0xc4202ec060, 0xa, 0xa, 0x0, 0x1, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590420, 0xc4202ec060, 0xa, 0xa, 0xa, 0xc4200a0c80, 0x0, 0xc42030a540)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590420, 0xc4202ec060, 0xa, 0xa, 0xc42064cf48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202ec060, 0xa, 0xa, 0x556616cf24a0, 0xc420590420, 0xc42064cf48, 0x2, 0x2, 0xc42064cfa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202ec040, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc420260000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a5a0, 0xc420590720, 0xc4201e4050, 0xc42030a540)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 102 [select, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run(0xc420590e40)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:191 +0x309
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.NewClient
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:62 +0x147

goroutine 97 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a208, 0x72, 0xc42035cb80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008bb18, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008bb18, 0xc42026c000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008bb00, 0xc42026c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008bb00, 0xc42026c000, 0x1000, 0x1000, 0xc42035cc68, 0x5566158d1f10, 0xc420310c00)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e6c0, 0xc42026c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420590ae0, 0xc4202ecb20, 0xa, 0xa, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420590ae0, 0xc4202ecb20, 0xa, 0xa, 0xa, 0xc4200a0c80, 0x0, 0xc4205e5da0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420590ae0, 0xc4202ecb20, 0xa, 0xa, 0xc42035cf48, 0x2, 0x2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.readMessageHeader(0xc4202ecb20, 0xa, 0xa, 0x556616cf24a0, 0xc420590ae0, 0xc42035cf48, 0x2, 0x2, 0xc42035cfa8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:54 +0x62
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*channel).recv(0xc4202ecb00, 0x556616d0d300, 0xc42003e040, 0x0, 0x2, 0xc420260000, 0xd6, 0xd6, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/channel.go:102 +0x6f
github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run.func1(0xc42030a660, 0xc420590e40, 0xc4201e4060, 0xc42030a600)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:164 +0x90
created by github.com/containerd/containerd/vendor/github.com/containerd/ttrpc.(*Client).run
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/ttrpc/client.go:159 +0x13f

goroutine 103 [syscall, 26 minutes]:
syscall.Syscall6(0xe8, 0x1f, 0xc42035d9b8, 0x80, 0xffffffffffffffff, 0x0, 0x0, 0x2f636f72702f202f, 0x622f73662f737973, 0x696d5f746d666e69)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/syscall/asm_linux_amd64.s:44 +0x5
github.com/containerd/containerd/vendor/golang.org/x/sys/unix.EpollWait(0x1f, 0xc42035d9b8, 0x80, 0x80, 0xffffffffffffffff, 0x746f727078616d2c, 0x657269642c353d6f, 0x5f657069702c7463)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1520 +0x7c
github.com/containerd/containerd/metrics/cgroups.(*oomCollector).start(0xc4202c5180)
    /home/travis/gopath/src/github.com/containerd/containerd/metrics/cgroups/oom.go:114 +0x7d
created by github.com/containerd/containerd/metrics/cgroups.newOOMCollector
    /home/travis/gopath/src/github.com/containerd/containerd/metrics/cgroups/oom.go:50 +0xf6

goroutine 105 [sleep]:
time.Sleep(0x2540be400)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/time.go:102 +0x16c
github.com/containerd/containerd/runtime/restart/monitor.(*monitor).run(0xc42056e298, 0x2540be400)
    /home/travis/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:155 +0x4c
created by github.com/containerd/containerd/runtime/restart/monitor.init.0.func1
    /home/travis/gopath/src/github.com/containerd/containerd/runtime/restart/monitor/monitor.go:85 +0x12c

goroutine 106 [semacquire, 26 minutes]:
sync.runtime_Semacquire(0xc42057720c)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/sema.go:56 +0x3b
sync.(*WaitGroup).Wait(0xc420577200)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/sync/waitgroup.go:129 +0x74
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*wgCloser).Wait(0xc4202b7040)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/helpers.go:63 +0x31
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Wait(0xc4202b6e80)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:224 +0x31
github.com/containerd/containerd.(*task).Delete(0xc420377260, 0x556616d0d340, 0xc4202b9860, 0xc4208d80e0, 0x1, 0x1, 0xd424ce7, 0xed3bf14ae, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/task.go:291 +0x377
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).loadContainer(0xc4203c4fc0, 0x556616d0d340, 0xc4202b9860, 0x556616d18d20, 0xc420866ee0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/restart.go:279 +0xedd
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).recover(0xc4203c4fc0, 0x556616d0d380, 0xc4202881b0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/restart.go:80 +0x869
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).Run(0xc4203c4fc0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/service.go:181 +0x103
github.com/containerd/containerd/vendor/github.com/containerd/cri.initCRIService.func1(0x556616d1ec00, 0xc4203c4fc0, 0x556616d0d300, 0xc42003e018)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/cri.go:103 +0x3a
created by github.com/containerd/containerd/vendor/github.com/containerd/cri.initCRIService
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/cri.go:102 +0x647

goroutine 114 [semacquire, 26 minutes]:
sync.runtime_notifyListWait(0xc42003cb50, 0xc400000000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/sema.go:510 +0x10f
sync.(*Cond).Wait(0xc42003cb40)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/sync/cond.go:56 +0x82
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).next(0xc420288090, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:103 +0x7d
github.com/containerd/containerd/vendor/github.com/docker/go-events.(*Queue).run(0xc420288090)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:68 +0x60
created by github.com/containerd/containerd/vendor/github.com/docker/go-events.NewQueue
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/docker/go-events/queue.go:29 +0x128

goroutine 115 [select, 26 minutes]:
github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe.func3(0xc4202880f0, 0xc4201ccd60, 0xc42052e060, 0x556616d0d2c0, 0xc420301180, 0xc4205902a0)
    /home/travis/gopath/src/github.com/containerd/containerd/events/exchange/exchange.go:171 +0x148
created by github.com/containerd/containerd/events/exchange.(*Exchange).Subscribe
    /home/travis/gopath/src/github.com/containerd/containerd/events/exchange/exchange.go:165 +0x279

goroutine 107 [IO wait]:
internal/poll.runtime_pollWait(0x7f2796b8a138, 0x72, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4a18, 0x72, 0xc42034f000, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4a18, 0xffffffffffffff00, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4203a4a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:372 +0x1aa
net.(*netFD).accept(0xc4203a4a00, 0xc42001cd80, 0x55661587a409, 0xc4202a1fc0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202f2360, 0x5566158d2790, 0xc42001cdc8, 0xc42001cdd0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202f2360, 0x556616cea270, 0xc4202a1f40, 0x556616d0d380, 0xc4205bec90)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/unixsock.go:253 +0x4b
net/http.(*Server).Serve(0xc420390000, 0x556616d0bd40, 0xc4202f2360, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:2773 +0x1a7
net/http.Serve(0x556616d0bd40, 0xc4202f2360, 0x556616cf4000, 0xc4205be930, 0x556616cea4a8, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:2392 +0x75
github.com/containerd/containerd/services/server.(*Server).ServeDebug(0xc4205bfef0, 0x556616d0bd40, 0xc4202f2360, 0x18, 0xc420065f38)
    /home/travis/gopath/src/github.com/containerd/containerd/services/server/server.go:188 +0x1c8
github.com/containerd/containerd/services/server.(*Server).ServeDebug-fm(0x556616d0bd40, 0xc4202f2360, 0xc4202f2360, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:157 +0x40
github.com/containerd/containerd/cmd/containerd/command.serve.func1(0x556616d0bd40, 0xc4202f2360, 0xc4203795a0, 0x556616d0d300, 0xc42003e018, 0xc42067b520, 0x1a)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:185 +0x77
created by github.com/containerd/containerd/cmd/containerd/command.serve
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:183 +0x1be

goroutine 108 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f2796b8a068, 0x72, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4c98, 0x72, 0xc42003cd00, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4c98, 0xffffffffffffff00, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Accept(0xc4203a4c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:372 +0x1aa
net.(*netFD).accept(0xc4203a4c80, 0xc42000e458, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc4202f23c0, 0x5566158d2790, 0xc420522d88, 0xc420522d90)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc4202f23c0, 0x556616ce98d8, 0xc42039e700, 0x556616d15920, 0xc42000e458)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/unixsock.go:253 +0x4b
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).Serve(0xc42039e700, 0x556616d0bd40, 0xc4202f23c0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:544 +0x21c
github.com/containerd/containerd/services/server.(*Server).ServeGRPC(0xc4205bfef0, 0x556616d0bd40, 0xc4202f23c0, 0x18, 0xc420063f38)
    /home/travis/gopath/src/github.com/containerd/containerd/services/server/server.go:167 +0x68
github.com/containerd/containerd/services/server.(*Server).ServeGRPC-fm(0x556616d0bd40, 0xc4202f23c0, 0xc4202f23c0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:171 +0x40
github.com/containerd/containerd/cmd/containerd/command.serve.func1(0x556616d0bd40, 0xc4202f23c0, 0xc420379670, 0x556616d0d300, 0xc42003e018, 0xc42067b5c0, 0x1f)
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:185 +0x77
created by github.com/containerd/containerd/cmd/containerd/command.serve
    /home/travis/gopath/src/github.com/containerd/containerd/cmd/containerd/command/main.go:183 +0x1be

goroutine 121 [select, 26 minutes]:
io.(*pipe).Read(0xc4209c0a00, 0xc42039c000, 0x1000, 0x1000, 0x0, 0xc42020e300, 0x1)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc4209c60c0, 0xc42039c000, 0x1000, 0x1000, 0x55661588aaf5, 0xc42039c000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc4203fbe70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc4203fbe70, 0xa, 0xc420577278, 0xc420063c10, 0x5566158c10fa, 0xc420577278, 0xc4203fbc10)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc4203fbe70, 0x1000, 0x1000, 0xc42039c000, 0x1000, 0x1000, 0xc420063c80)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc4207e9440, 0x3e, 0x556616cfad40, 0xc4209c60c0, 0x7f2796b8cd98, 0xc4203060c0, 0x556616546345, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc4207e9440, 0x3e, 0xc4209c60c0, 0x7f2796b8cd98, 0xc4203060c0, 0x556616546345, 0x6, 0x4000, 0xc42052f020)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 117 [select, 26 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc42003ce00, 0x1, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x135
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc420387140)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x1be
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc420278000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0xd4
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0xdab

goroutine 118 [select, 26 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420278000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:867 +0x243
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:280 +0xdd0

goroutine 119 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89f98, 0x72, 0xc420521b30)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4203a4118, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4203a4118, 0xc42040c000, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4203a4100, 0xc42040c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4203a4100, 0xc42040c000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c028, 0xc42040c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420386420, 0xc42057e658, 0x9, 0x9, 0x30, 0x28, 0x556616c06a20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420386420, 0xc42057e658, 0x9, 0x9, 0x9, 0xc420521d68, 0x55661587a540, 0xc420521e17)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420386420, 0xc42057e658, 0x9, 0x9, 0x556615ce7f66, 0xc4207aa9c0, 0xc4207a0004)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc42057e658, 0x9, 0x9, 0x556616cf24a0, 0xc420386420, 0x0, 0x556600000000, 0xc4207aa9c0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42057e620, 0xc4207aa9c0, 0xc4207aa9c0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420278000, 0xc4207aa990, 0x556616ce9930)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:426 +0x7e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc42039e700, 0x556616d17540, 0xc420278000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:676 +0xed
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:631 +0x3e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4207a7560, 0xc42039e700, 0x7f2796c6e428, 0xc420278000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:639 +0x29
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:638 +0x5fb

goroutine 109 [select, 5 minutes]:
io.(*pipe).Read(0xc4202f9b30, 0xc42058b000, 0x1000, 0x1000, 0x0, 0x160, 0xb0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc42000e2c0, 0xc42058b000, 0x1000, 0x1000, 0xc4208ab320, 0x556616546345, 0xc42066b0e0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc42035fe70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc42035fe70, 0xa, 0x20, 0xc4205764c8, 0x6, 0x8, 0xc4205764d8)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc42035fe70, 0x1, 0x8, 0xc4200e0240, 0x88, 0x90, 0xc4200e0240)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc42066b0e0, 0x45, 0x556616cfad40, 0xc42000e2c0, 0x7f2796b8cd98, 0xc4208ab320, 0x556616546345, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc42066b0e0, 0x45, 0xc42000e2c0, 0x7f2796b8cd98, 0xc4208ab320, 0x556616546345, 0x6, 0x4000, 0xc42031ede0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 110 [select, 26 minutes]:
io.(*pipe).Read(0xc4202f9bd0, 0xc420338000, 0x1000, 0x1000, 0x0, 0xc42020e300, 0x1)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc42000e2d0, 0xc420338000, 0x1000, 0x1000, 0x55661588aaf5, 0xc420338000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc420363e70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc420363e70, 0xa, 0xc4205764f0, 0xc420794c10, 0x5566158c10fa, 0xc4205764f0, 0xc420363c10)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc420363e70, 0x1000, 0x1000, 0xc420338000, 0x1000, 0x1000, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc42066b0e0, 0x45, 0x556616cfad40, 0xc42000e2d0, 0x7f2796b8cd98, 0xc4208ab320, 0x55661654633f, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc42066b0e0, 0x45, 0xc42000e2d0, 0x7f2796b8cd98, 0xc4208ab320, 0x55661654633f, 0x6, 0x4000, 0xc42031ef60)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 111 [chan receive, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers.func2(0xc42031ede0, 0xc42031ef60, 0xc42066b0e0, 0x45, 0xc42000e2b8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:169 +0xff
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:167 +0x203

goroutine 132 [IO wait, 5 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89df8, 0x72, 0xc4201edc98)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4208e2018, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4208e2018, 0xc4208c0001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4208e2000, 0xc4208c0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc4208d8000, 0xc4208c0000, 0x8000, 0x8000, 0xc4201eddb0, 0x5566158a4dd6, 0xc4201edda0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc4208d8000, 0xc4208c0000, 0x8000, 0x8000, 0x15f, 0x8000, 0x15f)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc420097e00, 0xc4208c0000, 0x8000, 0x8000, 0x15f, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc4208ab380, 0x7f2796bcdc88, 0xc420097e00, 0xc4208c0000, 0x8000, 0x8000, 0x556616b70e00, 0x0, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc4208ab380, 0x7f2796bcdc88, 0xc420097e00, 0xc420097e00, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func1(0xc4208ae280, 0xc4204f50f0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:110 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:109 +0x73

goroutine 133 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89ec8, 0x72, 0xc4208bc498)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420596bf8, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc420596bf8, 0xc4208da001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420596be0, 0xc4208da000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc42056e488, 0xc4208da000, 0x8000, 0x8000, 0x7f2796bcc6c8, 0x0, 0xc4208bc5a0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc42056e488, 0xc4208da000, 0x8000, 0x8000, 0x7f2796bcc6c8, 0x556615887a01, 0x10100c420030000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc420097e60, 0xc4208da000, 0x8000, 0x8000, 0x8000, 0x8000, 0x556616b59980)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc4208ab3a0, 0x7f2796bcdc88, 0xc420097e60, 0xc4208da000, 0x8000, 0x8000, 0x556616b70e00, 0x0, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc4208ab3a0, 0x7f2796bcdc88, 0xc420097e60, 0xc420097e60, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func2(0xc4208ae280, 0xc4204f50f0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:122 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:121 +0xd0

goroutine 139 [select, 26 minutes]:
io.(*pipe).Read(0xc420596e60, 0xc420248000, 0x1000, 0x1000, 0x0, 0xc42020e300, 0x1)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc4208d8068, 0xc420248000, 0x1000, 0x1000, 0x55661588aaf5, 0xc420248000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc420074e70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc420074e70, 0xa, 0xc42010a920, 0xc4208bcc10, 0x5566158c10fa, 0xc42010a920, 0xc420074c10)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc420074e70, 0x1000, 0x1000, 0xc420248000, 0x1000, 0x1000, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc4209c4050, 0x43, 0x556616cfad40, 0xc4208d8068, 0x7f2796b8cd98, 0xc420754420, 0x556616546345, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc4209c4050, 0x43, 0xc4208d8068, 0x7f2796b8cd98, 0xc420754420, 0x556616546345, 0x6, 0x4000, 0xc4204141e0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 135 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc4208ae600, 0x1, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x135
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc4207be960)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x1be
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc4205e6800)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0xd4
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0xdab

goroutine 136 [select, 26 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc4205e6800)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:867 +0x243
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:280 +0xdd0

goroutine 137 [IO wait]:
internal/poll.runtime_pollWait(0x7f2796b89d28, 0x72, 0xc4201efb30)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42008ac98, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc42008ac98, 0xc4208e4000, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42008ac80, 0xc4208e4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc42008ac80, 0xc4208e4000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42000e318, 0xc4208e4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420097f20, 0xc420538e38, 0x9, 0x9, 0x20, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420097f20, 0xc420538e38, 0x9, 0x9, 0x9, 0xc42f86f9a8, 0x2f86f9a81587a540, 0x5c2e6eaa)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420097f20, 0xc420538e38, 0x9, 0x9, 0xbf03b98aaf86f9a8, 0x16f1fd6a7f2, 0x556617492b20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc420538e38, 0x9, 0x9, 0x556616cf24a0, 0xc420097f20, 0x0, 0x556600000000, 0x2ccf7, 0xc4202f9d28)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420538e00, 0xc420320280, 0xc420320280, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4205e6800, 0xc4208a6b40, 0x556616ce9930)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:426 +0x7e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc42039e700, 0x556616d17540, 0xc4205e6800)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:676 +0xed
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:631 +0x3e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4208ab680, 0xc42039e700, 0x7f2796c6e428, 0xc4205e6800)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:639 +0x29
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:638 +0x5fb

goroutine 48 [select]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc420926040, 0x1, 0x0, 0x0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x135
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc420932000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x1be
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc420928000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0xd4
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0xdab

goroutine 49 [select, 26 minutes]:
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc420928000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:867 +0x243
created by github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:280 +0xdd0

goroutine 146 [IO wait]:
internal/poll.runtime_pollWait(0x7f2796b89c58, 0x72, 0xc420663b30)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4208fc098, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4208fc098, 0xc420906000, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4208fc080, 0xc420906000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc4208fc080, 0xc420906000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc4208d8008, 0xc420906000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
bufio.(*Reader).Read(0xc420904000, 0xc42091e038, 0x9, 0x9, 0x20, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:216 +0x23a
io.ReadAtLeast(0x556616cf24a0, 0xc420904000, 0xc42091e038, 0x9, 0x9, 0x9, 0xc411d12271, 0x11d122711587a540, 0x5c2e6eb2)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:309 +0x88
io.ReadFull(0x556616cf24a0, 0xc420904000, 0xc42091e038, 0x9, 0x9, 0xbf03b98c91d12271, 0x170def720a5, 0x556617492b20)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:327 +0x5a
github.com/containerd/containerd/vendor/golang.org/x/net/http2.readFrameHeader(0xc42091e038, 0x9, 0x9, 0x556616cf24a0, 0xc420904000, 0x0, 0x556600000000, 0x2f090, 0xc4208e2108)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
github.com/containerd/containerd/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42091e000, 0xc42024a720, 0xc42024a720, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
github.com/containerd/containerd/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420928000, 0xc4209240f0, 0x556616ce9930)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/transport/http2_server.go:426 +0x7e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc42039e700, 0x556616d17540, 0xc420928000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:676 +0xed
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:631 +0x3e
github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4208fe080, 0xc42039e700, 0x7f2796c6e428, 0xc420928000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:639 +0x29
created by github.com/containerd/containerd/vendor/google.golang.org/grpc.(*Server).handleRawConn
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:638 +0x5fb

goroutine 140 [select, 26 minutes]:
io.(*pipe).Read(0xc420596f00, 0xc42031d000, 0x1000, 0x1000, 0x1, 0x0, 0x1d)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc4208d8098, 0xc42031d000, 0x1000, 0x1000, 0x55661588aaf5, 0xc42031d000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc420079e70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc420079e70, 0xa, 0xc42010a938, 0xc4204f1c10, 0x5566158c10fa, 0xc42010a938, 0xc420079c10)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc420079e70, 0x1000, 0x1000, 0xc42031d000, 0x1000, 0x1000, 0x1000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc4209c4050, 0x43, 0x556616cfad40, 0xc4208d8098, 0x7f2796b8cd98, 0xc420754420, 0x55661654633f, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc4209c4050, 0x43, 0xc4208d8098, 0x7f2796b8cd98, 0xc420754420, 0x55661654633f, 0x6, 0x4000, 0xc420414360)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 141 [chan receive, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers.func2(0xc4204141e0, 0xc420414360, 0xc4209c4050, 0x43, 0xc4208d8058)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:169 +0xff
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:167 +0x203

goroutine 162 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89b88, 0x72, 0xc420065c98)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420597008, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc420597008, 0xc4207b4001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420596ff0, 0xc4207b4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc4208d80c0, 0xc4207b4000, 0x8000, 0x8000, 0x7f2796c6bdf8, 0x0, 0xc420065da0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc4208d80c0, 0xc4207b4000, 0x8000, 0x8000, 0x7f2796c6bdf8, 0x556615887a01, 0x10100c420030000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc4200976e0, 0xc4207b4000, 0x8000, 0x8000, 0x8000, 0x8000, 0x556616b59980)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc4207544e0, 0x7f2796bcdc88, 0xc4200976e0, 0xc4207b4000, 0x8000, 0x8000, 0x556616b70e00, 0xc4207aa900, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc4207544e0, 0x7f2796bcdc88, 0xc4200976e0, 0xc4200976e0, 0xc420278000, 0xc420065fb0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func1(0xc42034f880, 0xc42010a840)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:110 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:109 +0x73

goroutine 163 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89ab8, 0x72, 0xc4208b9498)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420597058, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc420597058, 0xc4206f6001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420597040, 0xc4206f6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc4208d80c8, 0xc4206f6000, 0x8000, 0x8000, 0x7f2796c6b808, 0x0, 0xc4208b95a0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc4208d80c8, 0xc4206f6000, 0x8000, 0x8000, 0x7f2796c6b808, 0x556615887a01, 0x10100c420030000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc420097740, 0xc4206f6000, 0x8000, 0x8000, 0x8000, 0x8000, 0x556616b59980)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc420754520, 0x7f2796bcdc88, 0xc420097740, 0xc4206f6000, 0x8000, 0x8000, 0x556616b70e00, 0x0, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc420754520, 0x7f2796bcdc88, 0xc420097740, 0xc420097740, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func2(0xc42034f880, 0xc42010a840)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:122 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:121 +0xd0

goroutine 122 [select, 4 minutes]:
io.(*pipe).Read(0xc4209c0aa0, 0xc4201e1000, 0x1000, 0x1000, 0x0, 0x140, 0x94)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:50 +0x117
io.(*PipeReader).Read(0xc4209c60d0, 0xc4201e1000, 0x1000, 0x1000, 0xc4203060c0, 0x55661654633f, 0xc4207e9440)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/pipe.go:127 +0x4e
bufio.(*Reader).fill(0xc42064be70)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:100 +0x120
bufio.(*Reader).ReadSlice(0xc42064be70, 0xa, 0x20, 0xc4205000b8, 0x6, 0x8, 0xc4205000d8)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:341 +0x2e
bufio.(*Reader).ReadLine(0xc42064be70, 0x1, 0x8, 0xc4206381c0, 0x6c, 0x70, 0xc4206381c0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/bufio/bufio.go:370 +0x36
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.redirectLogs(0xc4207e9440, 0x3e, 0x556616cfad40, 0xc4209c60d0, 0x7f2796b8cd98, 0xc4203060c0, 0x55661654633f, 0x6, 0x4000)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:91 +0x329
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger.func1(0xc4207e9440, 0x3e, 0xc4209c60d0, 0x7f2796b8cd98, 0xc4203060c0, 0x55661654633f, 0x6, 0x4000, 0xc42052f1a0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:58 +0x8c
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.NewCRILogger
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/logger.go:57 +0x2f3

goroutine 123 [chan receive, 26 minutes]:
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers.func2(0xc42052f020, 0xc42052f1a0, 0xc4207e9440, 0x3e, 0xc4209c60b8)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:169 +0xff
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).createContainerLoggers
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/container_start.go:167 +0x203

goroutine 128 [IO wait, 26 minutes]:
internal/poll.runtime_pollWait(0x7f2796b89918, 0x72, 0xc420792c98)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4200b2928, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4200b2928, 0xc4208ca001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4200b2910, 0xc4208ca000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc42056e0e0, 0xc4208ca000, 0x8000, 0x8000, 0x7f2796bcc598, 0x0, 0xc420792da0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc42056e0e0, 0xc4208ca000, 0x8000, 0x8000, 0x7f2796bcc598, 0x556615887a01, 0x10100c420030000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc4202b9b00, 0xc4208ca000, 0x8000, 0x8000, 0x8000, 0x8000, 0x556616b59980)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc420306180, 0x7f2796bcdc88, 0xc4202b9b00, 0xc4208ca000, 0x8000, 0x8000, 0x556616b70e00, 0x0, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc420306180, 0x7f2796bcdc88, 0xc4202b9b00, 0xc4202b9b00, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func1(0xc4202b6e80, 0xc420577200)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:110 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:109 +0x73

goroutine 129 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7f2796b899e8, 0x72, 0xc420520c98)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4209c0ba8, 0x72, 0xffffffffffffff01, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc4209c0ba8, 0xc42095e001, 0x8000, 0x8000)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4209c0b90, 0xc42095e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
os.(*File).read(0xc4209c60f8, 0xc42095e000, 0x8000, 0x8000, 0xc420520db0, 0x5566158a4dd6, 0xc420520da0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file_unix.go:226 +0x50
os.(*File).Read(0xc4209c60f8, 0xc42095e000, 0x8000, 0x8000, 0x6d, 0x8000, 0x6d)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/os/file.go:107 +0x6c
github.com/containerd/containerd/vendor/github.com/containerd/fifo.(*fifo).Read(0xc4202b9b60, 0xc42095e000, 0x8000, 0x8000, 0x6d, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/fifo/fifo.go:154 +0x220
io.copyBuffer(0x556616cf2c20, 0xc4203061c0, 0x7f2796bcdc88, 0xc4202b9b60, 0xc42095e000, 0x8000, 0x8000, 0x556616b70e00, 0x0, 0x7f2796bcdc88)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:400 +0x166
io.Copy(0x556616cf2c20, 0xc4203061c0, 0x7f2796bcdc88, 0xc4202b9b60, 0xc4202b9b60, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/io/io.go:362 +0x5c
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe.func2(0xc4202b6e80, 0xc420577200)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:122 +0x85
created by github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Pipe
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:121 +0xd0

goroutine 3629 [IO wait]:
internal/poll.runtime_pollWait(0x7f2796b89848, 0x72, 0xc4204f1e58)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc420930118, 0x72, 0xffffffffffffff00, 0x556616cf58a0, 0x55661744c0f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0x9d
internal/poll.(*pollDesc).waitRead(0xc420930118, 0xc420695900, 0x1, 0x1)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc420930100, 0xc420695901, 0x1, 0x1, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/internal/poll/fd_unix.go:157 +0x17f
net.(*netFD).Read(0xc420930100, 0xc420695901, 0x1, 0x1, 0x556616314e7c, 0xc4204f1f70, 0x5566158a44ab)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc42038c2a0, 0xc420695901, 0x1, 0x1, 0x0, 0x0, 0x0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/net.go:176 +0x6c
net/http.(*connReader).backgroundRead(0xc4206958f0)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:668 +0x5c
created by net/http.(*connReader).startBackgroundRead
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/net/http/server.go:664 +0xd0
$ sudo crictl pods
FATA[0000] listing pod sandboxes failed: rpc error: code = Unknown desc = server is not initialized yet

Seems deadlock is here in waiting for container IO to close?

goroutine 106 [semacquire, 26 minutes]:
sync.runtime_Semacquire(0xc42057720c)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/runtime/sema.go:56 +0x3b
sync.(*WaitGroup).Wait(0xc420577200)
    /home/travis/.gimme/versions/go1.10.5.linux.amd64/src/sync/waitgroup.go:129 +0x74
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*wgCloser).Wait(0xc4202b7040)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/helpers.go:63 +0x31
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io.(*ContainerIO).Wait(0xc4202b6e80)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/io/container_io.go:224 +0x31
github.com/containerd/containerd.(*task).Delete(0xc420377260, 0x556616d0d340, 0xc4202b9860, 0xc4208d80e0, 0x1, 0x1, 0xd424ce7, 0xed3bf14ae, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/task.go:291 +0x377
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).loadContainer(0xc4203c4fc0, 0x556616d0d340, 0xc4202b9860, 0x556616d18d20, 0xc420866ee0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/restart.go:279 +0xedd
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).recover(0xc4203c4fc0, 0x556616d0d380, 0xc4202881b0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/restart.go:80 +0x869
github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server.(*criService).Run(0xc4203c4fc0, 0x0, 0x0)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/pkg/server/service.go:181 +0x103
github.com/containerd/containerd/vendor/github.com/containerd/cri.initCRIService.func1(0x556616d1ec00, 0xc4203c4fc0, 0x556616d0d300, 0xc42003e018)
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/cri.go:103 +0x3a
created by github.com/containerd/containerd/vendor/github.com/containerd/cri.initCRIService
    /home/travis/gopath/src/github.com/containerd/containerd/vendor/github.com/containerd/cri/cri.go:102 +0x647

containerd 1.2.1 kubernetes 1.11.6 Ubuntu 18.04

This is a different environment than my other issue #1014.

Random-Liu commented 5 years ago

@steven-sheehy What is your systemd service file?

This usually happens because some processes in the container are leaked holding the container io. We fixed a bug related to this in 1.2.1, and it shouldn't happen again.

However, if you are not configuring systemd correctly, it may move container processes out of its original cgroup. In that case, some cleanup logic of runc will be broken, which may cause process leakage.

steven-sheehy commented 5 years ago

@Random-Liu It's the same one that comes with the release tarball.

$ cat /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStartPre=/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

My kubelet args if it helps:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-log-max-files=2 --container-log-max-size=100Mi --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --runtime-request-timeout=15m
steven-sheehy commented 5 years ago

systemctl status shows all the processes in /system.slice/containerd.service cgroup

$ systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-01-03 19:54:41 UTC; 2h 1min ago
     Docs: https://containerd.io
  Process: 242537 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 242538 (containerd)
    Tasks: 296 (limit: 4915)
   CGroup: /system.slice/containerd.service
           ├─ 11093 sleep 5
           ├─ 95283 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/fbd8d3488cb4fb18706df86d1aa94c2a2fff737b16ffb44062576191c974f1cf -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 95300 /pause
           ├─ 95374 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/cafb2a6f981903dff30007c3329b9f4c29b452bbe6ab4fddc0738adee3c6cd28 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 95539 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/3963deae9861bd99f1fde35819ecbf8642098e03a742bd1fab7f316495645f85 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 95561 /pause
           ├─ 95669 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/fab4e9c2bcadf1a652fe840cdc4a87c46d08e856ce3568bb8ed9e7f2c18ed350 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 95683 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/e399107ecc5c9705e1fd1a84ab32be2ac369e7d4fd1f1df41a479d956903f25f -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 95708 /pause
           ├─ 96014 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/9d0374d05c5de01b86fb620c846b9878dd17c8ccfb6e233a847e6e373b39b362 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96032 /pause
           ├─ 96092 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/5e467b900655fa9bfe5157d329b61b9b86d8fdec5e6a18a081365df69ea633b6 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96218 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/264a6a17b38e19a5bcc5c7b18b078efe5b103a596d4340a67bf18cb16d77bc41 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96251 /pause
           ├─ 96435 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/013d12bc9156460a60795ef4d571dba4c96a57e3ae2f3b76beec57cacd604727 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96454 /bin/sh -c sfacctd -f /etc/pmacct/sflow_replicator.conf -F /var/run/pmacct/sfacctd_tee.pid; sfacctd -f /etc/pmacct/sflow_receiver.conf -F /var/run/pmacct/sfacctd.pid
           ├─ 96467 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/e2d1a384e7b31bfca54d60ee1e7829699a5251fc0ae3cfc1794072f0183f8533 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96489 /pause
           ├─ 96529 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/616dcf6c787c956a34f93f19f907d0d59a1b9d003afed92a2ca86bf327c54137 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/container
           ├─ 96561 /pause
...

Should I be using kubelet with --cgroup-driver as systemd or cgroupfs? I think since I don't set it manually it uses cgroupfs.

Random-Liu commented 5 years ago

@steven-sheehy If that is the case, this should be another issue.

Let me check another thing could go wrong.

Random-Liu commented 5 years ago

@steven-sheehy

1) We need to first find which container causes this issue, and check its status.

Can you do ctr -n=k8s.io containers ls | grep -v pause to list all containers? Containerd always return containers in a sorted order, 3 containers have been successfully loaded, so the 4th one should be the bad one. We need the container id.

2) We need to understand why the io stuck for the container after it is stopped.

If I understand correctly, there should be some processes belong to the container left running. We need to find those processes and check their state.

a) We can first check whether runc still see the container /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io list. If runc still sees it, we can use /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io state ${CONTAINER_ID} to get some debug information. b) Check the container cgroup to see what is inside. You can run find /sys/fs/cgroup | grep ${CONTAINER_ID} to find the cgroup, and then check cgroup.procs to see what processes are inside. c) If you know what the container is doing, it would be helpful to find all processes inside the container e.g. with pstree.

With these information we should be able to find which process is left running, and figure out why it is not stopped.

If there are no processes leaked, that is even more weird...

steven-sheehy commented 5 years ago

Thanks for the detailed steps. Hopefully I got this right:

sudo ctr -n=k8s.io containers ls | grep -v pause | head -n 5
CONTAINER                                                           IMAGE                                                                      RUNTIME
013d12bc9156460a60795ef4d571dba4c96a57e3ae2f3b76beec57cacd604727    registry.gitlab.com/firescope/stratis/pmacct/edge-pmacct:4.0.0-rc1         io.containerd.runtime.v1.linux
0f65efaa107732926ac247073b21ea8f012666d2f312d5726ddce96e8b4ea126    sha256:097dd2748285f27a6049caeb404bf5cbeaed8ce5c5fb71f734cf6f125abc9fe6    io.containerd.runtime.v1.linux
10b309d5d2945eee4cabe23346a56ab3e7c037a1ff21e6d5646e98b5d3fdfbf0    sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e    io.containerd.runtime.v1.linux
14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841    sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e    io.containerd.runtime.v1.linux

$ sudo /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io list | grep 14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841   187411      running     /run/containerd/io.containerd.runtime.v1.linux/k8s.io/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841   2018-12-03T03:28:42.182943096Z   root

$ sudo /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io state 14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
{
  "ociVersion": "1.0.0",
  "id": "14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841",
  "pid": 187411,
  "status": "running",
  "bundle": "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841",
  "rootfs": "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/rootfs",
  "created": "2018-12-03T03:28:42.182943096Z",
  "annotations": {
    "io.kubernetes.cri.container-type": "sandbox",
    "io.kubernetes.cri.sandbox-id": "14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841"
  },
  "owner": ""

$ find /sys/fs/cgroup | grep 14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/pids.current
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/pids.max
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/pids.events
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/pids/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/hugetlb.2MB.failcnt
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/hugetlb.2MB.max_usage_in_bytes
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/hugetlb.2MB.usage_in_bytes
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/hugetlb/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/hugetlb.2MB.limit_in_bytes
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.sectors
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_serviced
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.time_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.read_bps_device
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.write_bps_device
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.weight_device
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_queued
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.write_iops_device
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_merged
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.time
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_service_bytes_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_wait_time
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.sectors_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_service_time_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.leaf_weight
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.weight
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_serviced_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_service_bytes
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_queued_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.leaf_weight_device
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.io_service_bytes
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.io_serviced
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.reset_stats
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_merged_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_service_time
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.io_wait_time_recursive
/sys/fs/cgroup/blkio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/blkio.throttle.read_iops_device
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.use_hierarchy
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.tcp.max_usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.slabinfo
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.tcp.usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.failcnt
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.force_empty
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.max_usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.event_control
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.swappiness
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.limit_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.pressure_level
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.max_usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.tcp.limit_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.stat
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.numa_stat
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.tcp.failcnt
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.oom_control
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.kmem.limit_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.soft_limit_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.failcnt
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.usage_in_bytes
/sys/fs/cgroup/memory/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/memory.move_charge_at_immigrate
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/freezer.parent_freezing
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/freezer.state
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/freezer/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/freezer.self_freezing
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.mems
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.sched_relax_domain_level
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.mem_exclusive
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.memory_pressure
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.cpus
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.mem_hardwall
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.memory_migrate
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.memory_spread_page
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.sched_load_balance
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.cpu_exclusive
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.effective_mems
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.effective_cpus
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuset.memory_spread_slab
/sys/fs/cgroup/cpuset/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/devices.list
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/devices.allow
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/devices/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/devices.deny
/sys/fs/cgroup/perf_event/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/perf_event/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/perf_event/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/perf_event/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/perf_event/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpu.stat
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.stat
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_percpu_sys
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_user
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_percpu_user
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_percpu
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpu.cfs_quota_us
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_sys
/sys/fs/cgroup/cpu,cpuacct/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cpuacct.usage_all
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/net_prio.ifpriomap
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/net_cls.classid
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs
/sys/fs/cgroup/net_cls,net_prio/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/net_prio.prioidx
/sys/fs/cgroup/systemd/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841
/sys/fs/cgroup/systemd/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.clone_children
/sys/fs/cgroup/systemd/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/tasks
/sys/fs/cgroup/systemd/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/notify_on_release
/sys/fs/cgroup/systemd/kubepods/pod6a6f0656-d640-11e8-a966-00505691e4af/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841/cgroup.procs

$ find /sys/fs/cgroup | grep 14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841 | grep "cgroup.procs" | xargs cat | uniq
187411

$ ps -aef | grep 187411
firesco+  26482 230337  0 23:30 pts/1    00:00:00 grep --color=auto 187411
root     187411 187386  0  2018 ?        00:00:00 /pause

$  ps -aef | grep 187386
firesco+  26345 230337  0 23:29 pts/1    00:00:00 grep --color=auto 187386
root     187386      1  0  2018 ?        00:01:21 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/14a2191736e10e8023b36f851b47ea758c227d1feef40ce94f003ef08f87f841 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/containerd
root     187411 187386  0  2018 ?        00:00:00 /pause

Looks like there's no container in this pod, just the pause. How do I find out what the pod name is?

Random-Liu commented 5 years ago

@steven-sheehy It seems that your pause image is sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e, you may want to grep -v sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e instead of grep -v pause.

Your current list includes the sandbox (pause container), that is why you see the /pause.

steven-sheehy commented 5 years ago

Let's try this again:

$ sudo ctr -n=k8s.io containers ls | grep -v pause | grep -v "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" | head -n 5
CONTAINER                                                           IMAGE                                                                      RUNTIME
013d12bc9156460a60795ef4d571dba4c96a57e3ae2f3b76beec57cacd604727    registry.gitlab.com/firescope/stratis/pmacct/edge-pmacct:4.0.0-rc1         io.containerd.runtime.v1.linux
0f65efaa107732926ac247073b21ea8f012666d2f312d5726ddce96e8b4ea126    sha256:097dd2748285f27a6049caeb404bf5cbeaed8ce5c5fb71f734cf6f125abc9fe6    io.containerd.runtime.v1.linux
290fb9bbf983bee091c3bf50676df4648319e347c38c9dcc533137e459be35b9    registry.gitlab.com/firescope/stratis/edge/edge-inventory:4.0.0-rc9        io.containerd.runtime.v1.linux
30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1    sha256:a5103f96993a0024428cd52663b85081163e66d8f24e7ac26a18555f432006b2    io.containerd.runtime.v1.linux

$ sudo /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io list | grep 30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1
30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1   0           stopped     /run/containerd/io.containerd.runtime.v1.linux/k8s.io/30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1   2018-12-03T03:28:30.827234075Z   root

$ sudo /usr/local/sbin/runc --root=/var/run/containerd/runc/k8s.io state 30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1
{
  "ociVersion": "1.0.0",
  "id": "30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1",
  "pid": 0,
  "status": "stopped",
  "bundle": "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1",
  "rootfs": "/run/containerd/io.containerd.runtime.v1.linux/k8s.io/30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1/rootfs",
  "created": "2018-12-03T03:28:30.827234075Z",
  "annotations": {
    "io.kubernetes.cri.container-type": "container",
    "io.kubernetes.cri.sandbox-id": "49539e8ac5a735c8946d683198957962ec0e186a9774541d8bb7c3fae49342ef"
  },
  "owner": ""
}

find /sys/fs/cgroup | grep 30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1 | grep cgroup.procs | xargs cat | sort -u
186522
186685

$ ps -aef | grep 186522
root     186522 186398  0  2018 ?        01:03:56 /home/weave/weaver --port=6783 --datapath=datapath --name=d2:44:84:04:ea:42 --host-root=/host --http-addr=127.0.0.1:6784 --metrics-addr=0.0.0.0:6782 --docker-api= --no-dns --db-prefix=/weavedb/weave-net --ipalloc-range=10.214.0.0/16 --nickname=qaedge1fce1-2 --ipalloc-init consensus=3 --conn-limit=100 --expect-npc 10.0.22.153 10.0.22.154 10.0.22.152

$ ps -aef | grep 186398
root     186398      1  0  2018 ?        00:01:20 containerd-shim -namespace k8s.io -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/k8s.io/30f2f8fd279d09abf726f7eacc0d180f43db06c6fc80680ca70e9777e564b8e1 -address /run/containerd/containerd.sock -containerd-binary /usr/local/bin/containerd
root     186522 186398  0  2018 ?        01:03:56 /home/weave/weaver --port=6783 --datapath=datapath --name=d2:44:84:04:ea:42 --host-root=/host --http-addr=127.0.0.1:6784 --metrics-addr=0.0.0.0:6782 --docker-api= --no-dns --db-prefix=/weavedb/weave-net --ipalloc-range=10.214.0.0/16 --nickname=qaedge1fce1-2 --ipalloc-init consensus=3 --conn-limit=100 --expect-npc 10.0.22.153 10.0.22.154 10.0.22.152
root     186685 186398  0  2018 ?        00:16:35 /home/weave/kube-utils -run-reclaim-daemon -node-name=qaedge1fce1-2 -peer-name=d2:44:84:04:ea:42 -log-level=debug

$ pstree 186398
containerd-shim─┬─kube-utils───21*[{kube-utils}]
                ├─weaver───32*[{weaver}]
                └─9*[{containerd-shim}]

So looks like it is Weave. Which makes sense since that is the last thing in the containerd logs

Random-Liu commented 5 years ago

@steven-sheehy Yeah, this is the problem.

You see that runc report the container is stopped (because the init process is dead). However, there are still processes running holding the io, which blocks the container delete.

Random-Liu commented 5 years ago

Did you upgrade containerd from some older version to 1.2.1? And the weave container was running before the update?

If that is the case, the running containerd-shim instance will be an old containerd-shim which doesn't include the fix https://github.com/containerd/containerd/pull/2854. And containerd won't be able to stop the container because of a known bug.

If this is confirmed, you may want to manually runc kill --all that container. We'll try to maintain backward compatibility and support in-place upgrade. However, 1) this is a bug in the old version, which is not a backward compatibility issue; 2) it is always recommended to drain your node before upgrading critical node components like kubelet, containerd.

And we may want to document this in the next release note as a known issue that requires action.

steven-sheehy commented 5 years ago

@Random-Liu That's probably the case. The weave container has a create date of 2018-12-03T03:28:30.712112913Z according to ctr and that PR was merged on 12/4.

When I upgraded containerd, I did drain the node using kubectl drain but it doesn't remove daemonsets like weave. If kubectl is not enough, should I be using crictl or ctr to drain? Do I need to stop kubelet since daemonsets always get rescheduled? I couldn't find any documentation on the proper procedure to upgrade containerd so I just did the above, but it sounds like it was not enough. Maybe this issue can be turned into a documentation issue where steps to upgrade between major, minor and bugfix releases are provided?

Thanks again for your help tracking down these issues.

Random-Liu commented 5 years ago

When I upgraded containerd, I did drain the node using kubectl drain but it doesn't remove daemonsets like weave.

I see. This is bad. :( We are about to release 1.2.2. We should include some instruction there.

This only happens when: 1) You upgrade containerd to 1.2+; 2) You have a pod not drained from the node (e.g. a daemonset), and the pod: a) Has a container contains multiple processes; b) Some processes don't exit with the container init process; c) The pod is using pod pid namespace.

If this is the case, you need to use runc kill --all to kill that container. :(

We'll simplify this a bit in the release note.

steven-sheehy commented 5 years ago

@Random-Liu Release notes to allow upgrades for this specific issue are good, but more generic upgrade instructions on https://containerd.io/docs/ or on github would be much appreciated as well. I'm still confused on exact commands, especially if you don't know what containers you need to runc kill.

Random-Liu commented 5 years ago

@steven-sheehy Does your weave pod has shareProcessNamespace: true in the pod spec?

steven-sheehy commented 5 years ago

No. For reference, this is the weave yaml we are applying.

Random-Liu commented 5 years ago

OK. So hostPID pod is also affected...

What containerd version did you upgrade from to 1.2.1?

steven-sheehy commented 5 years ago

To be honest, I'm not sure exactly. It was either 1.2.1-rc.0 or 1.2.0. We've only recently switched to containerd so those are the only versions we've ever used.

Random-Liu commented 5 years ago

@steven-sheehy I see.

I fully understand the problem now. The bug introduced by https://github.com/containerd/containerd/commit/b5ccc66c2c814cfc21d58678c854272427814b59 is soooooo bad... It only exists in 1.2 versions before 1.2.1, because of the bug, you can never stop a multi-process host/pod pid pod. I think we'll have to recommend a node REBOOT for the 1.2.x --> 1.2.1 update...

Random-Liu commented 5 years ago

And I'll add an integration test for this case to prevent future regression.

steven-sheehy commented 5 years ago

As for upgrade instructions, here are the steps I came up with when using Kubernetes and systemd. I'm not sure if these steps will suffice to avoid the deadlock when upgrading 1.2.0 to 1.2.1 or if a node reboot is required. But I think something like these should be added to an upgrade section of the documentation. Do these steps look correct?

kubectl drain $HOSTNAME --ignore-daemonsets --delete-local-data
systemctl stop kubelet.service
crictl stopp $(crictl pods -q)
systemctl stop containerd.service

# update containerd binary via package manager or manually

systemctl start containerd.service
systemctl start kubelet.service
Random-Liu commented 5 years ago

@steven-sheehy Thanks for the detailed instructions!

We currently don't have a place for update instructions. Release note seems to be a best place for this. We do similar thing for Kubernetes release note as well. :)

This has been included in the 1.2.2 release note https://github.com/containerd/containerd/releases/tag/v1.2.2.

Let's close this issue for now.