danielplohmann / mcrit

The MinHash-based Code Relationship & Investigation Toolkit (MCRIT) is a framework created to simplify the application of the MinHash algorithm in the context of code similarity.
GNU General Public License v3.0
86 stars 12 forks source link

Improve performance of job page #46

Closed danielplohmann closed 10 months ago

danielplohmann commented 1 year ago

The jobs page is currently very slow when having a large queue. One reason for this is certainly that the queue is barely using any indices on fields but also that rendering the page carries out multiple full collection scans and data retrievals as JobResource is doing the start/limit and filtering instead of having the database/MongoDB do this efficiently.

In order to improve performance for this,

danielplohmann commented 1 year ago

commit 435dc22 should now provide the foundation to address several of these things in the frontend.

danielplohmann commented 1 year ago

As a first step, this has now been integrated in v1.2.8, and should provide much better performance. Looks like this: image

The next step would be an extension with filtering for lists of sample_id and family_id (for rapid selection in the sample table) and exposing queue deletion functionality in the frontend. @yankovs feel free to test if the job page is more responsive now for your collection.

yankovs commented 1 year ago

Wow, this looks amazing! Great job :)

Our huge MCRIT instance is unfortunately down in favor of a slimmer one with much less samples. But i'll still test it out, maybe even on a local instance with lots of samples.