search

MetaCell / geppetto-scidash

Geppetto scidash extension
2 stars 1 forks source link

Improve performance of filters and score tests page when we have more than a few hundreds records #384

Open gidili opened 5 years ago

rgerkin commented 4 years ago

I'm not sure what performance looks like now (i.e. how it scales across the number of records). Is the main performance issue in retrieving them from the server? Because I would think that, once retrieved, even thousands of records should be filterable and pageable client-side fairly quickly.

Currently, is everything about each record (e.g. all model, test, and score fields) loaded when the record collection is loaded from the server? If so, can records be lazy-loaded? The records from each page (beyond page 1) could be completely loaded only when they first become visible in the browser (when the user goes to that page number or retrieves them by filter).

Whether any of this is worth it depends on the performance question above, and even then might be low priority until we actually hit this scenario (thousands of records).

ddelpiano commented 4 years ago

I'm not sure what performance looks like now (i.e. how it scales across the number of records). Is the main performance issue in retrieving them from the server? Because I would think that, once retrieved, even thousands of records should be filterable and pageable client-side fairly quickly.

Not an issue retrieving them, from what I recall this is due to the filters' code and interaction between different filters that sometimes lead to ridiculous amount of time to move to a different score page or give a different input in the filters and get the records back. It would need more time to investigate in the issue to exactly understand what is the problem but I noticed this when I pushed few hundred records (3-400) in the UI.

Currently, is everything about each record (e.g. all model, test, and score fields) loaded when the record collection is loaded from the server? If so, can records be lazy-loaded? The records from each page (beyond page 1) could be completely loaded only when they first become visible in the browser (when the user goes to that page number or retrieves them by filter).

I would need to check the code to see if this is not already the case but Griddle can definitely be configured to get the data lazy loaded as suggested.

gidili commented 4 years ago

Would agree that this is about optimising lazy loading of what's in the filter range. Right now whatever comes from the filters is showed into griddle. Might even be an issue of making the custom react components "lighter" in rendering, because we have other projects where we are loading 1000s of records into griddle and it's performing ok, so there is a component of investigation included in this estimate.