search

vozlt / nginx-module-vts

Nginx virtual host traffic status module
BSD 2-Clause "Simplified" License
3.17k stars 456 forks source link

High CPU usage #251

Open dwMiguelM opened 1 year ago

dwMiguelM commented 1 year ago

Hey folks,

Giving more attention to this issue (issues/221), due to the exact same fact, but let me show you in real-environment.

This test was made by generating 500kr/s (500.000 connections per second), the connections were generated using 15k unique IPs. Currently, the VTS configuration has 17 filter keys. Be aware, those filter keys catch everything that happens in the web server, they are heavily updated.

Heavy load without VTS: https://prnt.sc/_M-8xNbPPTk1 Heavy load with VTS: https://prnt.sc/PEQYLxSi5vE6

The VTS configurations/keys and further detailed information about this performance impact caused by the VTS can be provided to a VTS contributor/developer in a private conversation.

I hope we can work together to optimize this :)

Best Regards!

vozlt commented 1 year ago

@dwMiguelM Thanks for testing and reporting. If you test by gradually reducing the number of filters, won't the load be reduced? Right? As mentioned here, I know that the higher the number of filters, the lower the performance due to _ngx_shmtxlock(). To improve this, it seems to be necessary to periodically update the shared memory after collecting statistics from individual nginx workers. This feature is currently being considered.

dwMiguelM commented 1 year ago

@dwMiguelM Thanks for testing and reporting. If you test by gradually reducing the number of filters, won't the load be reduced? Right? As mentioned here, I know that the higher the number of filters, the lower the performance due to _ngx_shmtxlock(). To improve this, it seems to be necessary to periodically update the shared memory after collecting statistics from individual nginx workers. This feature is currently being considered.

I'm sorry for asking, but I would need to request you something, is it possible to get any contact yours?

Thank you in advance :)

vozlt commented 1 year ago

@dwMiguelM Email: vozlt.x+github@gmail.com

testn commented 1 year ago

@vozlt this sounds similar to this one right? https://github.com/Kong/kong-plugin-prometheus/commit/fd844dc292e88613763a77b540a2fcf036190c08

u5surf commented 1 year ago

@testn Yes, It seems that it fits what @vozlt said. lua-resty-counter can provide the individual worker counter. As far as I can see the PR that module use it to count per request and can sync to shm periodically. I seems that it is not easy to implement in this module the same mechanism of lua-resty-counter.