Open Sigfried opened 1 year ago
This context may be useful for evaluating at what size we want to consider that concept sets are "too large" and need this filtering.
However, I think the better approach is to do some manual testing in the wild and see how long csets take to load (if they load at all) at various sizes, on various different people's computers, and set a default for filtering out large csets based on that.
@Sigfried At the DL meeting this Thursday, the main discussion was between Harold and I and it was on this topic. I edited your original post. Please take a look when you're back.
Additional discussion for "4. General hiding of concepts / subtrees".
Steps
Difficulties Problem is that when you remove some of these things, you don't remove links between relevant parts of the tree. E.g. (a: normal node) -> (b: 0-patient concept) -> (c: normal node). If we remove 'b', we need to actually add it back because it has a child that is a normal node. Should only remove it if it is a leaf, or if all of its children are also to be removed.
Implementation If you hide 100 0-patient concepts, add a stub concept that indicates that you did that, and allows user to add back in.
Some notes on where Siggie was at in terms of progress and questions, based on our Scrum meeting yesterday:
performance and testing
Hope and Siggie put this together in regards to testing: https://docs.google.com/spreadsheets/d/1awyFHCbmr32mR9f1d7i0qIMHi17PESkTlVs58isEECI/edit#gid=1494465654
We looked at the above sheet today. Siggie suspects that when you compare csets but then clear selected csets, memory does not go down. If this is so, is it because our cache is using memory and not disk? Do we want to change that in any way?
We decided that different tests should have different timeout thresholds, some of them being longer than 1 minute because that's how long it takes to render the comparison.
We decided to see qualitatively what's going on when we load these large csets. We tried this as an example: https://icy-ground-0416a040f.2.azurestaticapps.net/cset-comparison?codeset_ids=909552172 It over 75k expressions, without includeDescendants, so same number of members. On Siggie's computer, it failed with "Oh Snap!" after about 1-2 minutes. On Joe's computer, after about a minute, the dialogue popped up with "page unresponsive; wait or exit?". It was laggy. Joe could not open up the developer console or change the zoom of the windows. Joe chose wait. Joe's page quit with "Aw, Snap!" after about 3-4 minutes.
Perhaps one issue is that DataCache & localStorage interaction. We have a DataCache object that gets stuff if you haven't already gotten it before. Otherwise gives from cache. Every time get new data, needs to write out the whole cache again to localStorage. Involves uncompress/compress and JSON stringify/parse.
Things Siggie will try to complete some of these, and take a stab at others:
The rest of the strategies after that will take some time to implement.
We decided to create a test_n3c_no_rxnorm
schema.
Lots of relevant crossover concerns with the octopus graph (see October 6 notes).
Overview
When there are too many concepts in a cset, it causes problems.
Goals:
Implementation
See frontend testing documentation
Sub-task list
Involves investigation strategies to try to find out the nature of the problem, as well as implementing some of the "optimization options". Sorted in the order we plan to try them.
Figure out what's causing performance and scaling problems
615
Optimization options
Done: