louislam / uptime-kuma

A fancy self-hosted monitoring tool
https://uptime.kuma.pet
MIT License
52.19k stars 4.7k forks source link

Uptime-Kuma causing ping floods #4837

Open MAMedici opened 3 weeks ago

MAMedici commented 3 weeks ago

โš ๏ธ Please verify that this question has NOT been raised before.

๐Ÿ›ก๏ธ Security Policy

๐Ÿ“ Describe your problem

I've had problems with uptime-kuma triggering IPS blocks due to excessive/abusive traffic. This latest incident I was able to talk to a network engineer who sent me the logs, Looking at the logs I see my IP is pinging them almost every second, instead of 300s as configured in uptime-kuma for this host.

I ran a packet capture on my firewall and indeed see that uptime-kuma is flooding several hosts with unexpectedly-high ICMP traffic.

I double-checked my site and general configuration settings, and everything loose as it should. Most of my entries are set for 5m (300s).

Any idea what might be causing this and how to make it act as expected? This is Uptime-Kuma 1.23.13 on docker 26.10 on a Debian 12 container on Proxmox. Nothing noteworthy appears in the Uptime-Kuma logs. Firewall logs and capture attached.

๐Ÿ“ Error Message(s) or Log

uptime-kuma ping capture.txt

๐Ÿป Uptime-Kuma Version

1.23.13

๐Ÿ’ป Operating System and Arch

Debian 12/Linux 6.5.13-5-pve/amd64

๐ŸŒ Browser

Firefox 126.0.1

๐Ÿ–ฅ๏ธ Deployment Environment

Runtime: Docker 26.10 Host: lxc Virtualization running Debian GNU/Linux 12 (bookworm) Linux 6.5.13-5-pve Architecture: x86-64 Database: sqlite Filesystem used to store the database on: number of monitors:

CommanderStorm commented 3 weeks ago

Cannot read the capture you send. Does somewhat look like the wrong interface was being captured, as having proxmox usually means running on a server and those usually don't run via wifi..

image

Have also tried looking at it in a text editor, but that looks like random data to me without an explaination

How many nodes do you have pinging away at said hosting provider with the firewall?

Btw: 1 ping/s = 1 packet/s I don't know why your provider considers this as flooding of multiple hosts. If I were a Provider, I would not worry about that.

CommanderStorm commented 3 weeks ago

Also could you link to the docs of your IPS?

I am assuming they don't outright block ICMP at 1 ping/s as

CommanderStorm commented 3 weeks ago

I have done a bit of digging. Here are the statistics from your capture assuming the second colllumn is the time in ยตs since the start. delta s = time since the last ping of the same ip1 and ip2

image

Without additional context what you are actually doing, I don't know if anything is misbehaving

MAMedici commented 3 weeks ago

Cannot read the capture you send. Does somewhat look like the wrong interface was being captured, as having proxmox usually means running on a server and those usually don't run via wifi..

Sorry, that was a text file that included the logs sent to me from the hosting provider's IPS (top section), and copy-paste from Wireshark of the pcap from my firewall. I've enclosed the pcap in the attached zip file.

How many nodes do you have pinging away at said hosting provider with the firewall?

I am pinging from one node on my end to one IP at the hosting provider.

Btw: 1 ping/s = 1 packet/s I don't know why your provider considers this as flooding of multiple hosts. If I were a Provider, I would not worry about that.

Not flooding multiple hosts at the hosting provider. But looking at the pcap, I can see Uptime Kuma is sending multiple pings several times to multiple hosts, at a rate of around 7-8/s. My expectation (and desire) is that Uptime Kuma would send one ping per defined testing interval for each monitored host/ip.

packetcapture-igb0-20240607155040.zip

MAMedici commented 3 weeks ago

Without additional context what you are actually doing, I don't know if anything is misbehaving

Attached are the settings for 104.225.208.3

![firefox_bVuuSqsjJw](https://github.com/louislam/uptime-kuma/assets/121360684/892b8950-b215-4de5-8744-3212b753228a)

CommanderStorm commented 3 weeks ago

104.225.208.3

Assuming you mean 104.225.208.23. That monitor had 1 ping in the capture noted above from 74.XX.XX.XX. In the timeframe, of 190s, that is reasonable. See the diagram in https://github.com/louislam/uptime-kuma/issues/4837#issuecomment-2155781333

There are a few cases in which we send a retry:

In the 189 seconds of traffic that you have approximately the amount of pings that I would expect.

Could you retry this in a less noisy environment and report the results? (docker run ..., then pcap and report the results) It is currently quite hard to guage if there is a problem or if this is just expected behaviour from my end..

MAMedici commented 5 days ago

I've temporarily removed uptime-kuma from my network as I'm working on another project atm. I'll circle back when I have time to devote my attention.