"/tmp/docker -c /tmp/k.conf" process use 100% CPU

[zong400]

• [x] This is a bug report
• [ ] This is a feature request
• [ ] I searched existing issues before opening this one

Steps to reproduce the behavior

process use 100% CPU and can not stop.

top:
top - 18:09:21 up  2:37,  4 users,  load average: 8.15, 4.83, 2.55
Tasks: 221 total,   3 running, 218 sleeping,   0 stopped,   0 zombie
%Cpu(s): 55.5 us, 23.4 sy,  0.0 ni, 21.0 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8043956 total,  3426532 free,  2318324 used,  2299100 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  5377592 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                  
 4278 root      20   0  406464  17288   1364 S  99.7  0.2   7:03.38 /tmp/docker -c /tmp/k.conf                               
 5255 dockeri+  20   0  406464  11180    852 S  99.7  0.1   4:00.13 /tmp/docker -c /tmp/k.conf                               
 5618 root      20   0    1992    108     64 R  59.1  0.0   1:13.95 6jmeijmed2meiria mp/k.conf                               
 4975 root      20   0    1992    112     64 R  50.2  0.0   2:47.45 kane8aneecne c /tmp/k.conf 

ps:
 1336  4189  4189  1336 ?           -1 Sl       0   0:00      \_ docker-containerd-shim -namespace moby -workdir /var/lib/dock
 4189  4224  4224  4224 ?           -1 Ssl      0   0:01      |   \_ registry serve /etc/docker/registry/config.yml
 4189  4278  4278  4278 ?           -1 Ssl      0   8:02      |   \_ /tmp/docker -c /tmp/k.conf

 1336  3661  3661  1336 ?           -1 Sl       0   0:00      \_ docker-containerd-shim -namespace moby -workdir /var/lib/dock
 3661  3683  3683  3683 ?           -1 Ss    1000   0:00      |   \_ /sbin/tini -- /usr/local/bin/jenkins.sh
 3683  3751  3751  3683 ?           -1 Sl    1000   0:33      |   |   \_ java -Duser.home=/var/jenkins_home -jar /usr/share/je
 3661  5255  5255  5255 ?           -1 Ssl   1000   4:59      |   \_ /tmp/docker -c /tmp/k.conf

 1336  2870  2870  1336 ?           -1 Sl       0   0:00      \_ docker-containerd-shim -namespace moby -workdir /var/lib/dock
 2870  2903  2903  2903 ?           -1 Ssl      0   0:11      |   \_ /coredns -conf /etc/coredns/Corefile
 2870  5617  5617  5617 ?           -1 Ss       0   0:00      |   \_ 6jmeijmed2meiria mp/k.conf
 5617  5618  5617  5617 ?           -1 R        0   1:46      |       \_ 6jmeijmed2meiria mp/k.conf

Output of docker version:

Client: Version: 18.03.1-ce API version: 1.37 Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:17:20 2018 OS/Arch: linux/amd64 Experimental: false Orchestrator: swarm

Server: Engine: Version: 18.03.1-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:15:30 2018 OS/Arch: linux/amd64 Experimental: false

Output of docker info:

Containers: 40 Running: 22 Paused: 0 Stopped: 18 Images: 38 Server Version: 18.03.1-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88 runc version: 4fc53a81fb7c994640722ac585fa9ca548971871 init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-91-generic Operating System: Ubuntu 16.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 7.671GiB Name: k8s-node1.bxr.cn ID: EHIM:VEHY:774Y:TYMX:YAT4:KH4E:FLYN:POF6:Y4LB:6RQI:QUMI:MZYW Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: registry.bxr.cn 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

k8s 1.10.3 ubuntu 16.04 4 core 8G mem

[nrupendesai]

I am having same issue. Is there any bug fix went in or any docker version that i can use which has fix above issue?

[leonunix]

same with me. The pod in my cluster will use all resource

[nadavami]

This is not a bug in docker, but a malicious process running a cryptocurrency miner. In other words, the system running this has been compromised and is being used to mine monero.

See below for more info.

You can recover the binary from memory using this example in StackExchange. Running a Symantec scan on the resulting binary identifies it as Linux.Coinminer.

I'll post more info when I get a chance to do more digging.

Update 1: It looks like the binary contains xmrig a monero miner & the -c /tmp/k.conf is the path to it's config file.

[dbinnersley]

This is an issue with an insecure deployment of kubernetes. Make sure there is no public access to the kubelet api. If port 10250 is exposed to the public on any nodes, then malicious processes will be able to execute commands in pods running on these nodes. I've noticed the same issue recently until public access was removed.

There is a good blog post here on Medium explaining the ramifications of this.

[superbogy]

same with me。