search

zecure / shadowd_ui

The Shadow Daemon user interface
GNU General Public License v2.0
20 stars 9 forks source link

Performance degrades greatly with just 1000 requests logged #44

Closed figassis closed 3 years ago

figassis commented 4 years ago

I got shadowd integrated into my wordpress sites yesterdat, and today, with 1051 requests logged in the UI, loading pages like /requests and /parameters loads for a while and then just times out. /profiles loads fine. So I'm assuming pagination is not being done properly?

figassis commented 4 years ago

It gets faster if I set my page size to 10. I also confirmed my database easily displays hundreds of rows in milliseconds, so not sure what's happening.

zit-hb commented 4 years ago

Pagination was not an issue there so far. For example, the demo runs on a low-end virtual machine and there are now more than 50,000 logged requests. The performance could certainly be improved but it is acceptable: https://demo.shadowd.zecure.org

It is possible to enable a debug mode in the UI. This adds the Symfony toolbar that contains a page to see which part takes so long. It's not intended to be used in the Docker containers though since the debug mode is a huge security risk.

I would recommend to run docker stats first to see which container has a high CPU usage when opening the pages.

Is the learning mode enabled by any chance? If there are many requests at the same time there is a lot of data to store in the database. If yes, try to disable it.

figassis commented 4 years ago

I'll go ahead and enable debug mode. Yes, the profile is in learning mode, there are about 36K requests in the db, but querying them directly form the db takes under a second. I would fix, but I'm not very experienced in symphony.

zit-hb commented 4 years ago

Okay, I would highly recommend to disable the learning mode. 36k requests in learning mode are already quite a lot. This will probably resolve your performance problems. shadowd might still continue adding requests to the database from an internal queue, so it will probably not be an instant change.