Slow concept tree display in SKOS vocabularies with large numbers of topConcept

[galviset]

(Issue found in the EarthPortal, using code from this repository up to October 2023) When loading a SKOS vocabulary that have a large amount of topConcept (the vocabulary is just a huge list of terms without hierarchy), the loading time of the concept tree becomes unreasonable.

A prime example would be this one : https://earthportal.eu/ontologies/NVS_P07/?p=classes

Using Firefox tools show a tremendous amount of AJAX queries with each fetching the label of a single term :

[syphax-bouazzouni]

I could not access NVS_P07 because maybe it is private.

But the same behavior can be reproduced with https://earthportal.eu/ontologies/NVS_L22/?p=classes, By disabling the cache in the developer console, and ordering the request by duration from the longest to the shortest, here what we can see

Problematics requests

Get concept labels

Multiple calls to https://earthportal.eu/ajax/classes/label?ontology=NVS_L22&id= , that takes from some 200ms to 20s and some do not resolve.

All the calls were made in https://earthportal.eu/ontologies/NVS_L22/?p=schemes, trying to get all the labels of the topConcept of the scheme mai scheme, in total 2064 requests.

But these requests are made asynchronously after the page load, so it is not the issue here even if we can do improvement here by fetching the label ones and not doing a request per concept.

Display the tree view

The real issue is in, the tree view request, https://earthportal.eu/ajax/classes/treeview?ontology=NVS_L22&conceptid=http%3A%2F%2Fvocab.nerc.ac.uk%2Fcollection%2FL22%2Fcurrent%2FTOOL0988%2F&concept_schemes=&auto_click=false, which takes 4 min to resolve

The function that handles this is; https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/controllers/concepts_controller.rb#L88-L90

Which calls this API endpoint https://earthportal.eu:8443/ontologies/NVS_L22/classes/roots, using the curl command below

curl -s -o /dev/null -w "%{time_total}\n"  -H "Cache-Control: no-cache" https://earthportal.eu:8443/ontologies/NVS_L22/classes/roots\?apikey\=c9147279-954f-41bd-b068-da9b0c44

it says that the API returns the roots, after 9 seconds, which is already a lot but it can not be the cause of a 4min response time. So the issue needs to be in the UI and further investigation needs to be done in the tree-view action (to be continued)

[galviset]

After benchmarking the function it seems the "faulty" line is this one:

https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/controllers/concepts_controller.rb#L100

All other lines of the show_tree function are executed in a fair amount of time.

[syphax-bouazzouni]

Can you go into that file, and check inside it what is taking time @galviset

[galviset]

Digging further leads to this loop in application_helper.rb which can take more than a second for each root concept :

https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/helpers/application_helper.rb#L172C1-L192

The line string << tree_link_to_concept(child: child, ontology_acronym: acronym, active_style: active_style, node: node, skos: !concept_schemes.nil?) is actually the one taking a lot of time.

I will investigate further the tree_link_to_concept method to see how it could be optimized

[syphax-bouazzouni]

I think the issue is these two lines https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/helpers/application_helper.rb#L199 => remove it

and https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/helpers/application_helper.rb#L202 => ontology_acronym should not be empty, if it is empty then there is an issue.

[galviset]

It was indeed these two lines. icons still need to be declared so I replaced it icons = nil I replaced the second line with href = "/ontologies/#{ontology_acronym}/concepts/?id=#{CGI.escape(child.id)}"

With these changes, NVS_L22 loads within 5 seconds on a local dev environment whereas it required several minutes before. If you agree with these changes, I'll do a pull request.

[syphax-bouazzouni]

you can make the pull request into ontoportal; the new release of Agroportal already integrates this fix (I think)

[galviset]

While these changes significantly reduces the loading time, the huge amount of AJAX requests is also responsible for loading time (and Firefox tab freeze) in production. More specifically, this function is the culprit : https://github.com/ontoportal-lirmm/bioportal_web_ui/blob/ba6ee8d863e7868ea3250745c3701f0685d6530b/app/helpers/application_helper.rb#L520-L531

Investigation continues.

[syphax-bouazzouni]

Here are some old issues discussing those Ajax calls, https://github.com/ncbo/bioportal-project/issues/184 and https://github.com/ncbo/bioportal-project/issues/181,

[syphax-bouazzouni]

related to https://github.com/agroportal/project-management/issues/360 and https://github.com/ontoportal/ontoportal-project/issues/54