search

dpriskorn / WikidataTopicCurator

Web application that helps Wikimedians add topics to items in Wikidata.
http://162.19.226.24:8080/
GNU Affero General Public License v3.0
0 stars 0 forks source link

bug: 502 error: gunicorn worker timeout when there are a lot of subtopics #31

Closed dpriskorn closed 7 months ago

dpriskorn commented 7 months ago

https://www.wikidata.org/wiki/Q11451 example check how many subtopics? how do we speed up the process?

dpriskorn commented 7 months ago

async and Wikibase REST API is the obvious solution.

Ainali commented 7 months ago

This one caused a 502: http://162.19.226.24:8080/check_subclass_of?qid=Q46952&lang=sv&subgraph=riksdagen_documents

Ainali commented 7 months ago

This one also: http://162.19.226.24:8080/check_subclass_of?qid=Q2990807&lang=sv&subgraph=riksdagen_documents

dpriskorn commented 7 months ago

I increased the gunicorn worker timeout to 200 seconds. It's not a good long term solution. The waiting is for getting labels and descriptions asynchronously from the Wikibase REST API. This tool is currently hosted in Europe far from the WMF datacenter so it takes a while when there are many subtopics.

dpriskorn commented 7 months ago

Oh, I might be wrong. image This indicates the worker crashes because of a bug. I'll have to test this more to find the cause. Alternatively just rewrite it using Javascript.

dpriskorn commented 7 months ago

Found a bug. Working on a fix.

dpriskorn commented 7 months ago

The bug is now fixed. image It still takes 30s to lookup labels, descriptions (async) and cirrussearch totals (no async) which is a pain for users. Under 2s would be ideal. I'm not sure I can do a lot about that because the waiting is all about latency. I can only lower the number of label+description requests by using SPARQL and that takes 3-5s at least on WDQS. Using SPARQL and the Freiburg QLever endpoint I might be able to reduce this to less than 5s total.

dpriskorn commented 7 months ago

Cut off 7 seconds by disabling the fetch for totals image

dpriskorn commented 7 months ago

I deployed the fixes above. Timeout is now 120 seconds.

dpriskorn commented 7 months ago

Both reported 502s above now work. Please report further 502s :)

dpriskorn commented 7 months ago

Hard test for new QLever SPARQL implementation: http://localhost:5000/check_subclass_of?qid=Q156&lang=en&subgraph=scientific_articles with 62 subtopics. Took 39s which is still far too long.

dpriskorn commented 7 months ago

Hooray! Because of caching in QLever the request time is now 1.4 seconds on a second run! image I absolutely love QLever!

dpriskorn commented 7 months ago

Deployed the new use-qlever branch. Please report any 502s.

dpriskorn commented 7 months ago

This extreme test with 3500+ subtopics now work in 4.4s http://162.19.226.24:8080/check_subclass_of?qid=Q378871&lang=en&subgraph=