Grouping: Optimize graph data download & display (AKA Very large cset crashes)

[Sigfried]

Overview

When there are too many concepts in a cset, it causes problems.

Goals:

1. Don't fetch so much data that it crashes or hobbles browser
2. Don't display so much data that crashes or hobbles browser

Implementation

See frontend testing documentation

Sub-task list

Involves investigation strategies to try to find out the nature of the problem, as well as implementing some of the "optimization options". Sorted in the order we plan to try them.

Figure out what's causing performance and scaling problems

• [x] Enhanced frontend testing

  • 615

• [ ] Find out which csets users are actually requesting and make performance tests for those. See: #590. This might be made easier if we have user login functionality. (And performance test for N3C recommended?)
• [ ] Investigate: Any front-end performance bottlenecks or issues due to bugs?
  • [ ] #617

    Optimization options

• [x] #658
• [x] #657
• [ ] Implement: "Hiding pre-set concept types (RxNorm extension)".
  • [ ] #547
  • [ ] #616
• [x] Try: "Don't cache very large concept sets"
  • [ ] #618
• [ ] Implement: "LRU (Last Recent Used) Cache"
  • [ ] #614
• [x] Implement: "General hiding of concepts / subtrees"
  • [ ] #602
  • If all else fails, this can fix our issue, but is the hardest / longest to implement.
• Pagination or dynamic collapsing (choose one)
  • [ ] #773
  • [ ] #772
• [ ] #662 (might split this up into multiple tasks)

Done:

• [x] 1.5. Are we making full use of all caching options? E.g.: Local storage, session storage Also look at: Increase cache size? Is this possible? Can we use a combination of local storage
• [x] 1. Test: Behavior w/out cache Deactivate the cache and see, how does the app behave? Does that solve some problems? Does that show us that the cache size is really our bottleneck? Will this experiment give us any ideas? Result: cuts time approximately in half. (Good but not enough.) See: Test results.

[joeflack4]

Context: Screenshots of cset counts frequency histograms

This context may be useful for evaluating at what size we want to consider that concept sets are "too large" and need this filtering. However, I think the better approach is to do some manual testing in the wild and see how long csets take to load (if they load at all) at various sizes, on various different people's computers, and set a default for filtering out large csets based on that.

[joeflack4]

@Sigfried At the DL meeting this Thursday, the main discussion was between Harold and I and it was on this topic. I edited your original post. Please take a look when you're back.

[joeflack4]

Discussion 2023/09/18 meeting

Additional discussion for "4. General hiding of concepts / subtrees".

Steps

1. Remove RxNorm
2. Remove 0-patient concepts
3. Collapse deepest levels

Difficulties Problem is that when you remove some of these things, you don't remove links between relevant parts of the tree. E.g. (a: normal node) -> (b: 0-patient concept) -> (c: normal node). If we remove 'b', we need to actually add it back because it has a child that is a normal node. Should only remove it if it is a leaf, or if all of its children are also to be removed.

Implementation If you hide 100 0-patient concepts, add a stub concept that indicates that you did that, and allows user to add back in.

[joeflack4]

Cache clearing options

• [ ] i. Older stuff expires
• [x] ii. Manually: Right now only way to empty is manually.
• [ ] iii. On schedule: Thought that we had implemented to clear itself regularly / daily, but looks like not so.

[joeflack4]

Some notes on where Siggie was at in terms of progress and questions, based on our Scrum meeting yesterday:

performance and testing

• (these are intertwined. want to improve performance but also don't know how it impacts our testing)
• playwright
  • there's a plugin
  • me: 2 timestamps suffice? just see how much time has elapsed between two UI manipulations?
  • when does it start to lag frame rate? how long until run out of memory?
• chrome / chromium has a performance API
  • this might be able to tell memory

[joeflack4]

Hope and Siggie put this together in regards to testing: https://docs.google.com/spreadsheets/d/1awyFHCbmr32mR9f1d7i0qIMHi17PESkTlVs58isEECI/edit#gid=1494465654

[joeflack4]

We looked at the above sheet today. Siggie suspects that when you compare csets but then clear selected csets, memory does not go down. If this is so, is it because our cache is using memory and not disk? Do we want to change that in any way?

We decided that different tests should have different timeout thresholds, some of them being longer than 1 minute because that's how long it takes to render the comparison.

We decided to see qualitatively what's going on when we load these large csets. We tried this as an example: https://icy-ground-0416a040f.2.azurestaticapps.net/cset-comparison?codeset_ids=909552172 It over 75k expressions, without includeDescendants, so same number of members. On Siggie's computer, it failed with "Oh Snap!" after about 1-2 minutes. On Joe's computer, after about a minute, the dialogue popped up with "page unresponsive; wait or exit?". It was laggy. Joe could not open up the developer console or change the zoom of the windows. Joe chose wait. Joe's page quit with "Aw, Snap!" after about 3-4 minutes.

Perhaps one issue is that DataCache & localStorage interaction. We have a DataCache object that gets stuff if you haven't already gotten it before. Otherwise gives from cache. Every time get new data, needs to write out the whole cache again to localStorage. Involves uncompress/compress and JSON stringify/parse.

Things Siggie will try to complete some of these, and take a stab at others:

1. Test: Behavior w/out cache
2. Implement: "3. Hiding pre-set concept types (RxNorm extension)". See https://github.com/jhu-bids/TermHub/issues/547
3. Investigate: Any bugs / performance bottlenecks? and maybe:
4. Try: "6. Don't cache very large concept sets"

The rest of the strategies after that will take some time to implement.

[joeflack4]

We decided to create a test_n3c_no_rxnorm schema.

Lots of relevant crossover concerns with the octopus graph (see October 6 notes).