search

gbv / cocoda

A web-based tool for creating mappings between knowledge organization systems.
https://coli-conc.gbv.de/cocoda/
MIT License
39 stars 5 forks source link

Improve performance for big data sets #208

Closed stefandesu closed 2 years ago

stefandesu commented 5 years ago

When working with large systems without hierarchy or after working in the application for a long time without reloading, i.e. every time a lot of items are stored in the Vuex store, there are performance issues that affect the whole application. The cause is likely Vue's reactivity for all the objects stored in Vuex (thousands of objects each with a bunch of properties that all have to be watched by Vue). Right now, the performance penalty is not too big (it's still usable, just a little slow), so this is not the highest priority, but long term we should aim to improve the performance for these cases.

Related: https://github.com/vuejs/vue/issues/4384

stefandesu commented 5 years ago

Performance was already drastically improved by recent refactoring (as a side effect), but certain things still slow down the application. What I noticed in particular is that both selecting and clearing a concept scheme with a large number of top concepts takes a long time. I would guess that it takes that long for Vue to render or remove the thousands of new nodes in ConceptTree. Ideally, ConceptTree would only render the nodes that are currently in view, but this is not a trivial thing to implement (though not impossible of course).

Edit: I was able to confirm this by making ConceptTree just show a subset of all items. In that case, even with vocabularies with 5000 top concepts, the loading times were insignificant.

stefandesu commented 4 years ago

Reopening for #513. Suggestion: Try to improve performance for large data sets by inlining ConceptListItem into ConceptList (because it's only used there).

stefandesu commented 4 years ago

Looking back at this issue, I think the main problem is the big number of Vue components that are created with big datasets. One step would be to inline as many components as possible (where it makes sense of course), as I mentioned in the previous comment. It would also be good to simplify the code of ConceptList and ConceptListItem. If it's possible, it would make sense to turn off certain functionality for items that are not currently visible.

stefandesu commented 4 years ago

If we use vue-virtual-scroller, we can keep using a proper component for ItemName, even in the concept lists, but have only the ones rendered that are actually shown. This should improve performance for long lists significantly.

stefandesu commented 4 years ago

I'm working on getting vue-virtual-scroller to work in Cocoda. Progress is slow because everything's pretty complicated. We will also lose the scrolling animation when a concept is selected (scrolling still works, but just not animated). But when this is properly implemented, we could have VERY long lists without performance issues.

stefandesu commented 4 years ago

I've been having more problems with this and opened an issue in vue-virtual-scroller for something that might not be possible to do (yet). So I'm putting this on hold. Maybe we have to reconsider and find a different solution.

stefandesu commented 4 years ago

Here's a custom virtual scroller implementation that seems to be doing exactly what we need: https://stackoverflow.com/questions/48515997/how-to-handle-vue-2-memory-usage-for-large-data-50-000-objects

Example: https://codepen.io/zupkode/pen/MWwgLyb

stefandesu commented 2 years ago

With #584, this isn't really an issue anymore and even lists with 50k concepts work fairly well. There's still room for improvement, especially when it comes to reactivity, but that will be a separate issue.