salopensource / sal

Modular reporting for Endpoints
Apache License 2.0
212 stars 64 forks source link

inventory optimization #447

Closed sheagcraig closed 1 year ago

sheagcraig commented 1 year ago

This PR attempts to deal with the inventory app's performance challenges at scale.

To test, I created a mock inventory database in increasing sizes until I was able to create a performance bottleneck. Average cold load of the application list view took about 3000ms. Searches take significantly longer, around 10s.

The (main) strategy here is that we remove the "install count" annotation from the queryset, which cuts the initial load time about in half. We also remove unnecessary queries to the settings table of the DB and instead just query it once at import time. This shaves off ~100ms additionally.

The big improvement comes to searching and column ordering. Simply enabling the django-datatable-view feature to use caching for the queryset cuts the search time from around 10kms to 300ms.

Additionally, apparently django-datatable-view is back under active development and has made all of the python3 changes needed to allow us to remove our forked version and just pip install it.

sheagcraig commented 1 year ago

I tossed a datatables update from 1.10.11 to 1.13.5 in here as well.