search

dirkjanm / ROADtools

A collection of Azure AD/Entra tools for offensive and defensive security purposes
MIT License
1.75k stars 249 forks source link

[bug] [feature request] No results when database is huge - introduce results paging #33

Open mgeeky opened 2 years ago

mgeeky commented 2 years ago

Hi there Dirk!

Thank you for a splendid tool! :-)

I've just collected a rather big database (2,5GB) containing more than 100k users, 500k groups, 260k devices etc. The gather phase went just fine, but when I attempted to review results from GUI - no records could be rendered.

Whenever frontend app launches Ajax request, the server is unable to serialize JSON in a timely manner, making frontend wait extremely long for the response. This results in GUI displaying no records until server responds.

In my situation, the server was unable to come up with a response in 30 minutes.

Maybe there could be a paging functionality added with frontend API calling out to specific results-pages as a workaround?

Leaving this as a food for thought! :-)

Regards, Mariusz.

dirkjanm commented 2 years ago

Hey,

I've run into this as well on large tenants. The problem here is that for all tenants except those extremely large ones it works fine to just load all the data and use in-memory filtering. If doing paging on the backend that would need to include searching and filtering as well. Which essentially means I have to rebuild a large part of the frontend and backend just to accomodate those larger tenants. This may be something I will do at some point, but at the moment it's not a high priority.

mgeeky commented 2 years ago

Yes, I agree - these changes would entail substantial upgrades to both frontend and backend. Yet I believe they would offer worth ROI while working in larger tenants. Currently I'm pretty much unable to efficiently explore data collected - and I'm yet to work with more of that large tenants in upcoming engagements.

Anyway, I'm keeping my fingers crossed for you to find time to address this one! :-)

Regards, M.

rgmz commented 1 week ago

If doing paging on the backend that would need to include searching and filtering as well. Which essentially means I have to rebuild a large part of the frontend and backend just to accomodate those larger tenants. This may be something I will do at some point, but at the moment it's not a high priority.

@dirkjanm I've created a POC in #94 (only pagination, sorting, and filtering for /api/users). Have a look and let me know your thoughts.