calab-ntu / gpu-cluster

Eureka and Spock GPU clusters
3 stars 0 forks source link

DOS_DNS issue #40

Closed xuanweishan closed 2 years ago

xuanweishan commented 2 years ago

Main Issue

We got block due to DOS_DNS.

Temporary Solution

  1. Stop port 53 and 5353 in all machine, including tumaz.
    • eureka
      1. Check if port 53 and 5353 available on the node: netstat –an | grep 53
      2. Disable the software which using port 5353.
        systemctl stop avahi-daemon
        systemctl stop avahi-daemon.socket
    • tuamz
      1. Check if port 53 and 5353 available. netstat –an | grep 53
      2. Disable the software which using port 53.
      3. Edit file /etc/systemd/resolved.conf DNSStubListener=no
      4. Restart resolved service
        systemctl daemon-reload
        systemctl restart systemd-resolved.service
  2. Open DoS protection on eater and ironman.
  3. Change root and admin password for every machine.
xuanweishan commented 2 years ago

Follow up

tumaz

  1. Found that tumaz can't resolve domains.

    This makes tumaz can not connect to internet.

    Solution: Set up the resolve server to 8.8.8.8 (google DNS server) by edit /etc/resolv.conf

    #nameserver 127.0.0.53     (Comment out this line to prevent resolve domain ourselves)
    nameserver 8.8.8.8         (Add this line to apply google DNS server to resolve domain)
  2. Update system.
    sudo apt update
    sudo apt upgrade
  3. Scan malwares
    1. Install ClamAV: apt install clamav
    2. Scan malwares: clamscan -r /

      Result: No malware was found.

  4. Turn off the port which allows other machine can ssh to tumaz directly.
    1. Login to the router.
    2. Turn off the virtual server.
      Advance setting > virtual server > uncheck tumaz in the list
  5. Set up auto update every month at crontab

eureka

  1. After some nodes reboot without setting avahi back to disable. But our IP wasn't blocked again. So it might not be the reason why we are blocked.
  2. Update system.
    yum --skip-broken -y update
  3. Set up auto update every month at crontab