digitalocean / marketplace-partners

Image validation, automation, and other tools for DigitalOcean Marketplace Vendors and Custom Image users
Other
193 stars 98 forks source link

Intrusion attempts during image build cause non-empty logs #90

Open ximon18 opened 4 years ago

ximon18 commented 4 years ago

It is possible that the Droplet is attacked between the invocation of the DigitalOcean Marketplace cleanup.sh which truncates the logs and img_check.sh scripts which checks the logs, resulting in img_check.sh warning about non-empty logs due to logged intrusion attempts, and thus failing the Packer build.

Or at least I assume that is what was happening in my case, see below. I could not otherwise explain the sudden appearance of firewall related log entries after the log had been cleared.

Stopping UFW isn't an option as then img_check.sh complains. Stopping UFW logging only prevents some logging, not all. Stopping rsyslog didn't help either.

My solution was to apply a DigitalOcean firewall to the Droplet created by Packer that allowed SSH only from my IP address, thereby preventing the unwanted communication attempts from reaching the VM and thus being logged.

Automating this is non-trivial as the Packer DigitalOcean builder doesn't support enabling a DO firewall on the Droplet, nor does Packer expose the Droplet ID to the template. In theory one can use the cloud-init query instance_id command in a shell provisioner to write the Droplet ID to a file, then download it using a file provisioner and then use a shell-local provisioner to execute doctl compute firewall create to create the firewall, and then delete the firewall at the end. However, Packer doesn't have the notion of steps to always run and so the teardown of the firewall will not happen if any of the prior build steps fail, and running shell-local commands assumes the host has doctl installed and configured with the right DO API token, and brittle assumptions about which command syntax will work on the host have to be made (is it Linux, is it Windows?).

If you have any advice on how to avoid this issue that would be most appreciated. Alternatively, extending the DigitalOcean Packer builder to be able to add a firewall to the created Droplet limiting SSH to "my ip" (perhaps via a Packer user variable) would help work around this problem.

Here is an example of the problem that I encountered:

img_check.sh warnings:

    digitalocean: [WARN] un-cleared log file, /var/log/auth.log found
    digitalocean: [WARN] un-cleared log file, /var/log/kern.log found
    digitalocean: [WARN] un-cleared log file, /var/log/ufw.log found

Actual log file contents at the time:

    digitalocean: /var/log/alternatives.log
    digitalocean: /var/log/apt
    digitalocean: /var/log/auth.log
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
    digitalocean: Feb  1 20:33:02 packer-vm-name sshd[2570]: fatal: No supported key exchange algorithms [preauth]
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
    digitalocean: Feb  1 20:33:16 packer-vm-name sshd[2575]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
    digitalocean: Feb  1 20:33:17 packer-vm-name sshd[2575]: fatal: No supported key exchange algorithms [preauth]
    digitalocean: /var/log/btmp
    digitalocean: /var/log/cloud-init-output.log
    digitalocean: /var/log/dist-upgrade
    digitalocean: /var/log/dpkg.log
    digitalocean: /var/log/journal
    digitalocean: /var/log/kern.log
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/landscape
    digitalocean: /var/log/lastlog
    digitalocean: /var/log/lxd
    digitalocean: /var/log/syslog
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:32:41 packer-vm-name systemd-resolved[664]: message repeated 2 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/tallylog
    digitalocean: /var/log/ufw.log
    digitalocean: Feb  1 20:33:02 packer-vm-name kernel: [   72.894358] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=45.134.179.15 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=249 ID=40724 PROTO=TCP SPT=42847 DPT=3400 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:33:24 packer-vm-name kernel: [   95.091120] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:8a:7d:08:00 SRC=89.248.168.41 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=36109 PROTO=TCP SPT=47977 DPT=1677 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:13 packer-vm-name kernel: [  144.845855] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=103.72.8.7 DST=188.166.18.107 LEN=44 TOS=0x00 PREC=0x00 TTL=243 ID=45136 PROTO=TCP SPT=58355 DPT=3395 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:34:32 packer-vm-name kernel: [  163.622002] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=117.148.157.48 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=27459 PROTO=TCP SPT=53965 DPT=1433 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:19 packer-vm-name kernel: [  210.565897] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=93.174.95.110 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=61464 PROTO=TCP SPT=47917 DPT=7394 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: Feb  1 20:35:51 packer-vm-name kernel: [  242.250986] [UFW BLOCK] IN=eth0 OUT= MAC=6a:1f:6d:9b:3d:41:f4:a7:39:d7:82:7d:08:00 SRC=185.39.10.124 DST=188.166.18.107 LEN=40 TOS=0x00 PREC=0x00 TTL=251 ID=11142 PROTO=TCP SPT=49649 DPT=17625 WINDOW=1024 RES=0x00 SYN URGP=0
    digitalocean: /var/log/wtmp
ximon18 commented 4 years ago

FYI my current way of automating the work around for this problem is to run the following command after launching Packer:

doctl compute droplet list --format "ID,Name" | fgrep packer | awk '{print $1}' | xargs doctl compute firewall add-droplets XXXX --droplet-ids

Where XXX is the ID of a firewall I created whose only rule is to permit SSH only from my public IP address.