Closed cmoulliard closed 4 years ago
cmoulliard
commented
4 years ago I added an IP rule on the existing VMs to block the outbound traffic to the port 2375
116.203.41.7
88.99.186.195Something that I dont understand is why calls occur to the port 2375 as locally the docker daemon port opened is 2376. This is perhaps not at all related of course
[root@k8s-116 ~]# cat /etc/docker/daemon.json
{
"hosts": ["unix://","tcp://0.0.0.0:2376"],As the machine has been hacked again beginning of Friday afternoon, I updated the Iptables
iptables -A OUTPUT -p tcp --destination-port 4244
iptables -A OUTPUT -p tcp --destination-port 4243Changed some rules as described here: https://linoxide.com/how-tos/linux-sysctl-tuning/
vi /etc/sysctl.conf
...
# ignore ICMP packets (ping requests)
net.ipv4.icmp_echo_ignore_all=1
# This "sanity checking" helps against spoofing attack.
net.ipv4.conf.all.rp_filter=1
# Syn Flood protection
net.ipv4.tcp_syncookies = 1I did a search to find new executables installed on 88.99.186.19 without success
[root@k8s-115 ~]# touch --date "2020-02-17" /tmp/start
[root@k8s-115 ~]# touch --date "2020-02-21" /tmp/end
[root@k8s-115 ~]# find / -type f -executable -newer /tmp/start -not -newer /tmp/end
find: '/proc/23067/task/23067/fdinfo/6': No such file or directory
find: '/proc/23067/fdinfo/5': No such file or directory
/var/lib/cloud/instances/4650188/scripts/runcmd
/var/lib/cni/networks/cbr0/lock
/var/lib/yum/repos/x86_64/7/kubernetes/gpgdir-ro/secring.gpg
/var/lib/yum/repos/x86_64/7/kubernetes/gpgdir-ro/pubring.gpg
/var/lib/yum/repos/x86_64/7/kubernetes/gpgdir-ro/gpg.conf
/var/lib/yum/repos/x86_64/7/kubernetes/gpgdir-ro/pubring.gpg~
/var/lib/yum/repos/x86_64/7/kubernetes/gpgdir-ro/trustdb.gpgI'm doing now a yum update
[root@k8s-115 ~]# yum update
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.plustech.de
* epel: ftp.plusline.net
* extras: mirror.plustech.de
* updates: ftp.plusline.net
Resolving Dependencies
--> Running transaction check
---> Package ansible.noarch 0:2.4.2.0-2.el7 will be updated
---> Package ansible.noarch 0:2.9.3-1.el7 will be an update
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package docker-ce.x86_64 0:18.06.2.ce-3.el7 will be updated
---> Package docker-ce.x86_64 3:19.03.6-3.el7 will be an update
--> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.6-3.el7.x86_64
--> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.6-3.el7.x86_64
---> Package kernel.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Processing Dependency: linux-firmware >= 20190429-72 for package: kernel-3.10.0-1062.12.1.el7.x86_64
---> Package kernel-tools.x86_64 0:3.10.0-1062.9.1.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1062.12.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-1062.9.1.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1062.12.1.el7 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package python-passlib.noarch 0:1.6.5-2.el7 will be obsoleted
---> Package python-perf.x86_64 0:3.10.0-1062.9.1.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-1062.12.1.el7 will be an update
---> Package python2-passlib.noarch 0:1.7.1-1.el7 will be obsoleting
---> Package sg3_utils.x86_64 0:1.37-18.el7_7.1 will be updated
---> Package sg3_utils.x86_64 0:1.37-18.el7_7.2 will be an update
---> Package sg3_utils-libs.x86_64 0:1.37-18.el7_7.1 will be updated
---> Package sg3_utils-libs.x86_64 0:1.37-18.el7_7.2 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package sudo.x86_64 0:1.8.23-4.el7_7.1 will be updated
---> Package sudo.x86_64 0:1.8.23-4.el7_7.2 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-sysv.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-sysv.x86_64 0:219-67.el7_7.3 will be an update
--> Running transaction check
---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed
---> Package docker-ce-cli.x86_64 1:19.03.6-3.el7 will be installed
---> Package linux-firmware.noarch 0:20190429-72.gitddde598.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===========================================================================================================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================================================================================================
Installing:
kernel x86_64 3.10.0-1062.12.1.el7 updates 50 M
python2-passlib noarch 1.7.1-1.el7 epel 741 k
replacing python-passlib.noarch 1.6.5-2.el7
Updating:
ansible noarch 2.9.3-1.el7 epel 17 M
binutils x86_64 2.27-41.base.el7_7.2 updates 5.9 M
curl x86_64 7.29.0-54.el7_7.2 updates 270 k
docker-ce x86_64 3:19.03.6-3.el7 docker 24 M
kernel-tools x86_64 3.10.0-1062.12.1.el7 updates 7.9 M
kernel-tools-libs x86_64 3.10.0-1062.12.1.el7 updates 7.8 M
libcurl x86_64 7.29.0-54.el7_7.2 updates 223 k
python-perf x86_64 3.10.0-1062.12.1.el7 updates 7.9 M
sg3_utils x86_64 1.37-18.el7_7.2 updates 646 k
sg3_utils-libs x86_64 1.37-18.el7_7.2 updates 65 k
sqlite x86_64 3.7.17-8.el7_7.1 updates 394 k
sudo x86_64 1.8.23-4.el7_7.2 updates 842 k
systemd x86_64 219-67.el7_7.3 updates 5.1 M
systemd-libs x86_64 219-67.el7_7.3 updates 411 k
systemd-sysv x86_64 219-67.el7_7.3 updates 88 k
Installing for dependencies:
containerd.io x86_64 1.2.10-3.2.el7 docker 23 M
docker-ce-cli x86_64 1:19.03.6-3.el7 docker 40 M
linux-firmware noarch 20190429-72.gitddde598.el7 base 73 M
Transaction Summary
===========================================================================================================================================================================================================
Install 2 Packages (+3 Dependent packages)
Upgrade 15 Packages
Total download size: 266 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/20): curl-7.29.0-54.el7_7.2.x86_64.rpm | 270 kB 00:00:00
(2/20): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:00
(3/20): docker-ce-cli-19.03.6-3.el7.x86_64.rpm | 40 MB 00:00:00
(4/20): kernel-3.10.0-1062.12.1.el7.x86_64.rpm | 50 MB 00:00:01
(5/20): ansible-2.9.3-1.el7.noarch.rpm | 17 MB 00:00:02
(6/20): kernel-tools-3.10.0-1062.12.1.el7.x86_64.rpm | 7.9 MB 00:00:00
(7/20): kernel-tools-libs-3.10.0-1062.12.1.el7.x86_64.rpm | 7.8 MB 00:00:00
(8/20): libcurl-7.29.0-54.el7_7.2.x86_64.rpm | 223 kB 00:00:00
(9/20): python-perf-3.10.0-1062.12.1.el7.x86_64.rpm | 7.9 MB 00:00:00
(10/20): sg3_utils-1.37-18.el7_7.2.x86_64.rpm | 646 kB 00:00:00
(11/20): sg3_utils-libs-1.37-18.el7_7.2.x86_64.rpm | 65 kB 00:00:00
(12/20): python2-passlib-1.7.1-1.el7.noarch.rpm | 741 kB 00:00:00
(13/20): sqlite-3.7.17-8.el7_7.1.x86_64.rpm | 394 kB 00:00:00
(14/20): sudo-1.8.23-4.el7_7.2.x86_64.rpm | 842 kB 00:00:00
(15/20): systemd-219-67.el7_7.3.x86_64.rpm | 5.1 MB 00:00:00
(16/20): systemd-libs-219-67.el7_7.3.x86_64.rpm | 411 kB 00:00:00
(17/20): systemd-sysv-219-67.el7_7.3.x86_64.rpm | 88 kB 00:00:00
(18/20): linux-firmware-20190429-72.gitddde598.el7.noarch.rpm | 73 MB 00:00:03
(19/20): binutils-2.27-41.base.el7_7.2.x86_64.rpm | 5.9 MB 00:00:05
(20/20): docker-ce-19.03.6-3.el7.x86_64.rpm | 24 MB 00:00:05
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 41 MB/s | 266 MB 00:00:06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
Updating : libcurl-7.29.0-54.el7_7.2.x86_64 1/36
Updating : systemd-libs-219-67.el7_7.3.x86_64 2/36
Updating : systemd-219-67.el7_7.3.x86_64 3/36
Installing : containerd.io-1.2.10-3.2.el7.x86_64 4/36
Updating : sg3_utils-libs-1.37-18.el7_7.2.x86_64 5/36
Installing : 1:docker-ce-cli-19.03.6-3.el7.x86_64 6/36
Updating : kernel-tools-libs-3.10.0-1062.12.1.el7.x86_64 7/36
Installing : linux-firmware-20190429-72.gitddde598.el7.noarch 8/36
Installing : kernel-3.10.0-1062.12.1.el7.x86_64 9/36
Updating : kernel-tools-3.10.0-1062.12.1.el7.x86_64 10/36
Updating : 3:docker-ce-19.03.6-3.el7.x86_64 11/36
Updating : sg3_utils-1.37-18.el7_7.2.x86_64 12/36
Updating : systemd-sysv-219-67.el7_7.3.x86_64 13/36
Updating : curl-7.29.0-54.el7_7.2.x86_64 14/36
Updating : ansible-2.9.3-1.el7.noarch 15/36
Updating : python-perf-3.10.0-1062.12.1.el7.x86_64 16/36
Installing : python2-passlib-1.7.1-1.el7.noarch 17/36
Updating : sqlite-3.7.17-8.el7_7.1.x86_64 18/36
Updating : sudo-1.8.23-4.el7_7.2.x86_64 19/36
Updating : binutils-2.27-41.base.el7_7.2.x86_64 20/36
Cleanup : ansible-2.4.2.0-2.el7.noarch 21/36
Cleanup : systemd-sysv-219-67.el7_7.2.x86_64 22/36
Cleanup : docker-ce-18.06.2.ce-3.el7.x86_64 23/36
Cleanup : systemd-219-67.el7_7.2.x86_64 24/36
Cleanup : kernel-tools-3.10.0-1062.9.1.el7.x86_64 25/36
Cleanup : curl-7.29.0-54.el7_7.1.x86_64 26/36
Cleanup : sg3_utils-1.37-18.el7_7.1.x86_64 27/36
Erasing : python-passlib-1.6.5-2.el7.noarch 28/36
Cleanup : sg3_utils-libs-1.37-18.el7_7.1.x86_64 29/36
Cleanup : libcurl-7.29.0-54.el7_7.1.x86_64 30/36
Cleanup : kernel-tools-libs-3.10.0-1062.9.1.el7.x86_64 31/36
Cleanup : systemd-libs-219-67.el7_7.2.x86_64 32/36
Cleanup : python-perf-3.10.0-1062.9.1.el7.x86_64 33/36
Cleanup : sqlite-3.7.17-8.el7.x86_64 34/36
Cleanup : sudo-1.8.23-4.el7_7.1.x86_64 35/36
Cleanup : binutils-2.27-41.base.el7_7.1.x86_64 36/36Something strange discovered on the machine 88.
The file ./.ssh/authorized_keys has changed this friday, 6 minutes before the attack
find . -type f -mtime -3 -exec ls -l {} \; > last3days.txt
cat last3days.txt
[root@k8s-115 ~]# cat last3days.txt
-rw------- 1 root root 769 Feb 21 13:58 ./.ssh/authorized_keysIf we run this script
touch aut-keys.sh
chmod +x aut-keys.sh
vi aut-keys.sh
#!/bin/bash
for X in $(cut -f6 -d ':' /etc/passwd |sort |uniq); do
if [ -s "${X}/.ssh/authorized_keys" ]; then
echo "### ${X}: "
cat "${X}/.ssh/authorized_keys"
echo ""
fi
done
./aut-keys.shwe got this result
### /home/centos:
ssh-rsa AAAAB3NzaC1y....Bbw45gyw9Gn8cXPT
### /opt/autoupdater:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
### /opt/logger:
ssh-rsa AAAAB3NzaC1y...Pw40gkfgr47b2IqLKG2OJv
### /opt/system:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
### /root:
ssh-rsa AAAAB3Nza......45gyw9Gn8cXPT
ssh-rsa AAAAB3NzaC1yc...aGVWc5c3kTdXds4w4w/oFow== bionicWhy such users have been able to log on (=> hetzner user, ...)
### /opt/autoupdater:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
### /opt/logger:
ssh-rsa AAAAB3NzaC1y...Pw40gkfgr47b2IqLKG2OJv
### /opt/system:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJvThe file
./.ssh/authorized_keyshas changed this friday, 6 minutes before the attack
@jacobdotcosta Can you open a ticket to Hetzner to ask them if they added their key within the authorized keys ssh file and if they add such users
### /opt/autoupdater:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
### /opt/logger:
ssh-rsa AAAAB3NzaC1y...Pw40gkfgr47b2IqLKG2OJv
### /opt/system:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
jacobdotcosta
commented
4 years ago I added an IP rule on the existing VMs to block the outbound traffic to the port
2375116.203.41.7 88.99.186.195Something that I dont understand is why calls occur to the port
2375as locally the docker daemon port opened is2376. This is perhaps not at all related of course[root@k8s-116 ~]# cat /etc/docker/daemon.json { "hosts": ["unix://","tcp://0.0.0.0:2376"],
Our machine is used as a zombie in attacks against other servers, and the attack is targeted to the default 2375 port on the target machines.
cmoulliard
commented
4 years ago Our machine is used as a zombie in attacks against other servers, and the attack is targeted to the default 2375 port on the target machines.
Different ports have been finally used and not only 2375. So this is not at all related to a docker flaw
cmoulliard
commented
4 years ago We need perhaps a role/script able to find what is described here after
1) To find unusual traffic where score is > 20-50
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
2) To see the `fromTCP:port` to `destinationTCP:port`
netstat -an
3) To find the process listening on the port detected at step 2)
lsof -i :4001WDYT ? @jacobdotcosta
jacobdotcosta
commented
4 years ago /opt/autoupdater:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
/opt/logger:
ssh-rsa AAAAB3NzaC1y...Pw40gkfgr47b2IqLKG2OJv
/opt/system:
ssh-rsa AAAAB3NzaC1yc...Pw40gkfgr47b2IqLKG2OJv
The ticket is created.
cmoulliard
commented
4 years ago To restrict the ssh authorized keys to be used with some from ip addresses, we could add such parameter : https://debian-administration.org/article/685/Restricting_SSH_logins_to_particular_IP_addresses
jacobdotcosta
commented
4 years ago We need perhaps a role/script able to find what is described here after
1) To find unusual traffic where score is > 20-50 netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n 2) To see the `fromTCP:port` to `destinationTCP:port` netstat -an 3) To find the process listening on the port detected at step 2) lsof -i :4001WDYT ? @jacobdotcosta
I think it's a good idea, to have a playbook that collects this information. This way we only have to launch a playbook.
jacobdotcosta
commented
4 years ago To restrict the ssh authorized keys to be used with some
fromip addresses, we could add such parameter : https://debian-administration.org/article/685/Restricting_SSH_logins_to_particular_IP_addresses
It's an interesting idea! Although I think it's difficult to implement due to our nature. IMO we could use this if we were working together in the same location, to allow only local connections. As this is a cloud server and we're a disperse team, I find this difficult to implement without locking ourselves out.
jacobdotcosta
commented
4 years ago Audit has failed to restart.
cmoulliard
commented
4 years ago Do we still have something to do here ? @jacobdotcosta. If everything has been done, please close it with your PR
jacobdotcosta
commented
4 years ago Do we still have something to do here ? @jacobdotcosta. If everything has been done, please close it with your PR
I think we're good with the latest playbook that executes all security roles.
Issue
As reported 2 times this week by Hetzner, our
centos vmgenerates outbound traffic to the following port number from ports501xxAction items
To be first discussed/challenged @jacobdotcosta
yum updateaudit: https://www.digitalocean.com/community/tutorials/how-to-use-the-linux-auditing-system-on-centos-7This "sanity checking" helps against spoofing attack.
net.ipv4.conf.all.rp_filter=1
Syn Flood protection
net.ipv4.tcp_syncookies = 1
lsof -Pni | egrep "(UDP|LISTEN)" chronyd 1057 chrony 5u IPv4 19921 0t0 UDP 127.0.0.1:323 chronyd 1057 chrony 6u IPv6 19922 0t0 UDP [::1]:323 dhclient 1325 root 6u IPv4 20914 0t0 UDP :68 dockerd 1610 root 6u IPv6 19375 0t0 TCP :2376 (LISTEN) sshd 1630 root 3u IPv4 23370 0t0 TCP :22 (LISTEN) sshd 1630 root 4u IPv6 23372 0t0 TCP :22 (LISTEN) ...