jcchavezs
commented
7 months ago Hi Michael,
Thanks for the detailed description. I think the first place to look at is if there is a memory leak somewhere. Maybe emitting go metrics at this point is the right thing to do. Then we can start looking deeper. Another option is to deploy coraza with go profiling enabled.
On Thu, 21 Dec 2023, 16:22 Michael Militzer, @.***> wrote:
Hi,
we would like to use coraza-spoa in production but unfortunately as soon as we put production traffic on it, after a few hours the coraza-spoa daemon "goes nuts", starts using up all available CPU resources and shows a rapidly increasing memory consumption up to consuming tens of GBs of RAM. The system then starts swapping and becomes entirely unresponsive until the OOM killer finally kills the coraza-spoa daemon. And this happens quite frequently for us, like several times a day.
We run multiple haproxy load-balancing servers and what I observed is that often when coraza-spoa on one "goes nuts" the issue propagates also to further load-balancers, so the coraza daemon also on other servers starts excessively using memory and cpu almost at the same time. My conclusion therefore would be that the problem is externally triggered, so caused by certain (malicious?) requests that come in. Only this would explain the time-correlation of the OOM event among multiple load-balancer servers.
However, I went through our access logs at the respective times and couldn't find any obviously unusual or suspicious requests. There also were not unusually many requests at those times, so it's not an issue with general load or request rate. But still, there must be something apparently, so certain problematic requests that trigger the issue and cause the excessive resource use. I also checked the coraza logs but there is absolutely nothing logged when it happens. So whatever it is, coraza apparently didn't attempt to block those requests (or wasn't even able to get that far at all in processing the request).
So I don't really have a handle on how to get down to the root of the problem. I don't even know whether the issue is in coraza itself or the problem is unique to the haproxy SPOE integration. So if you have any ideas what might be the problem or how we could debug it more effectively, please let me know.
Thanks! Michael
— Reply to this email directly, view it on GitHub https://github.com/corazawaf/coraza-spoa/issues/97, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYAUE3OSL4I7CE6DOMZLYKRH4ZAVCNFSM6AAAAABA6R6QO6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TENRSGIYTANI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
mmilitzer
NullIsNot0
Hi,
we would like to use coraza-spoa in production but unfortunately as soon as we put production traffic on it, after a few hours the coraza-spoa daemon "goes nuts", starts using up all available CPU resources and shows a rapidly increasing memory consumption up to consuming tens of GBs of RAM. The system then starts swapping and becomes entirely unresponsive until the OOM killer finally kills the coraza-spoa daemon. And this happens quite frequently for us, like several times a day.
We run multiple haproxy load-balancing servers and what I observed is that often when coraza-spoa on one "goes nuts" the issue propagates also to further load-balancers, so the coraza daemon also on other servers starts excessively using memory and cpu almost at the same time. My conclusion therefore would be that the problem is externally triggered, so caused by certain (malicious?) requests that come in. Only this would explain the time-correlation of the OOM event among multiple load-balancer servers.
However, I went through our access logs at the respective times and couldn't find any obviously unusual or suspicious requests. There also were not unusually many requests at those times, so it's not an issue with general load or request rate. But still, there must be something apparently, so certain problematic requests that trigger the issue and cause the excessive resource use. I also checked the coraza logs but there is absolutely nothing logged when it happens. So whatever it is, coraza apparently didn't attempt to block those requests (or wasn't even able to get that far at all in processing the request).
So I don't really have a handle on how to get down to the root of the problem. I don't even know whether the issue is in coraza itself or the problem is unique to the haproxy SPOE integration. So if you have any ideas what might be the problem or how we could debug it more effectively, please let me know.
Thanks! Michael