search

BroadSoft-Xtended / BroadWorks-Dashboards-and-Discovery

This repository contains the BroadWorks Dashboards and Discovery components to extend BroadWorks data mining, reporting, and analysis capabilities.
23 stars 4 forks source link

"TCP ZeroWindow" Between Sender and Receiver #87

Closed MistAssassin closed 4 years ago

MistAssassin commented 4 years ago

I am running into a situation where one of my AS servers is causing the receiver to respond with a "TCP ZeroWindow". This is the only server in the stack we have that does this. For reference, I have had 4 AS/NS pairs pointed that this receiver with no problems, but as soon as this particular AS gets pointed at a receiver, well it's not pretty. I'm hoping you guys can point in the right direction for some config to figure out why this is happening.

Receiver: 32 CPU | 8GB Ram | default config. Increased numthreads to 24 while troubleshooting. as15 [causing problems]: default sender config. same broadsoft version. same OS/kernel version. expecting double the other AS in terms of logging. as11 [no problems]: default sender config. same broadsoft version. Same OS/kernel version. ~2 million documents over 5 minutes. Just to reiterate, I've had 4 AS pairs of similar size as as11 pointed at this receiver with no problems.

When troubleshooting as15: all threads that the receiver is allowed to use, immediately go to 100% and stay that way. Load goes to w/e the value the receiver's configured at. Data comes in a small burst window, then the "ZeroWindow" happens, and nothing comes in for ~2 minutes. Rinse and repeat. Maybe 60k documents get created over 5 minutes.

When as11 is connected, traffic comes in normally. Load on the server is sub 1.

I attached a wireshark capture from the receiver with traffic from both AS servers. Any assistance would be greatly appreciated.

as wiresharks.zip

MistAssassin commented 4 years ago

Worked around this with Kafka. This AS seems to be producing about close to 100x the documents the others are. Will have to work on scaling the receivers in order to handle that.