Biobank explorer even slower on master

[marikaris]

How to Reproduce

Upload the data + meta data of the eric package (get it on accept server via navigator) THIS DOES WORK NOW, YAY! It almost makes up for this issue Then upload the most recent version of the app via the app manager (for instance this one: molgenis-app-biobank-explorer.zip) Add biobank explorer to menu Open network tab in inspect Go to biobank explorer

Expected behavior

It loads in += 8 seconds (as in 8.2.4). For anonymous this should be around 30 seconds (which is a complete other issue)

Observed behavior

It loads in 24-25 seconds. For anonymous this is += 40 seconds.

[dennishendriksen]

Analysis

With the current master deployed on localhost the biobank explorer loads in about three seconds on my laptop when signed in as a superuser:

Out of the 30 (!) requests required to load the page this can mainly be attributed to the request: GET http://localhost/api/v2/eu_bbmri_eric_biobanks?num=10000&attrs=collections(id,description,materials,diagnosis_available,name,type,order_of_magnitude(*),size,sub_collections(*),parent_collection,quality(*),data_categories),*

Which takes about 2.5 (!) seconds (anonymous: 14s): and results in 6.2MB (!) of data.

After the biobank explorer finished loading a table with pager is rendered displaying 10 rows with biobank label, juridical_person and collection types.

Since the current query retrieves all available biobank it is to be expected that the performance will degrade once more biobanks are added.

Solution

Possibly the same information could be produced with the following query:

Data API v2

GET /api/v2/eu_bbmri_eric_biobanks?attrs=id,name,juridical_person,collections(id,type(id,label))&sort=name&num=10&start=0

which takes about 25ms (~100x faster!) and results in 16KB (~400x smaller!) of data.

Data API v3

GET /api/data/eu_bbmri_eric_biobanks?filter=name,juridical_person,collections(type(label))&expand=collections(type)&sort=name&page=0&size=10

which takes about 20ms (~125x faster!) and results in 14KB (~450x smaller!) of data.

Selecting another page should then trigger a new query with a different start (v2) or page (v3) value. Selecting a row should trigger a query for that specific biobank.

[dennishendriksen]

master - admin: 2.5s, anonymous: 14s

1) The fix for '#8502 spring security beans are not using the molgenis rolehierarchy' degrades performance (see: 50fd93007596cc9b1c999ed4b695de09b8e910fb) 2) This performance degradation was reported in '#8695 slow performance retrieving data for non-superusers' 3) The degradation was addressed in molgenis/molgenis#8699 by resolving various other performance bottlenecks 4) Part of molgenis/molgenis#8699 is a 'fetch validator' that fixes incorrect fetches on-the-fly to prevent new runtime issues due to incorrect fixes in application or user code 5) Fetch validator/fixer can be a performance bottleneck for deep valid fetches 6) ... such as the fetch of the mega-bbmri-eric query

master-without-fetch-validator/fixer - admin: 1.8s, anonymous: 6.9s

[esthervanenckevort]

The changes how the app works make the performance acceptable again, issue can be closed.

[jelmerveen]

performance gain has been implemented, closing this ticket