search

zeek / broker

Zeek's Messaging Library
https://docs.zeek.org/projects/broker
Other
65 stars 28 forks source link

Potential issue with `put_unique` #319

Open Neverlord opened 1 year ago

Neverlord commented 1 year ago

Reported in the testing group: it seems like there's a potential performance issue with put_unique. Many writes from Zeek lead to an increasing memory usage over time until the system keels over.

Neverlord commented 1 year ago

To clarify, the actual report was about memory bloat. So far, I couldn't reproduce the bloat itself. What I did find is that Broker seems to do fine for throughput, but latency is quite high. This makes roundtrips for put_unique quite slow and Broker can only do ~4-5k put_unique per second with one writer/clone. Adding more writers/clones slows down further. I didn't try "interleaving" writes, though. The setup in the testing groups seems to be calling put_unique non-stop (without waiting for results first). That could explain the bloat if the system simply does more writes than Broker can handle.