Create person profiles - Githubissues

physikerwelt commented 10 months ago

Describe the issue

[x] find all persons via SPARQL


PREFIX wdt: <https://portal.mardi4nfdi.de/prop/direct/>
PREFIX wd: <https://portal.mardi4nfdi.de/entity/>

SELECT (COUNT(*) as ?Triples) WHERE { ?item wdt:P31 wd:Q57162}

- [x] Create list with Person, Person name
```SPARQL
PREFIX wdt: <https://portal.mardi4nfdi.de/prop/direct/>
PREFIX wd: <https://portal.mardi4nfdi.de/entity/>
SELECT ?item ?itemLabel
WHERE { ?item wdt:P31 wd:Q57162 .
   SERVICE wikibase:label {
     bd:serviceParam wikibase:language "en" .
   }
} 
LIMIT 5 OFFSET 1000

[x] Make list iterable in block sizes that don't generate timeouts

done via subquery. This was actually a novel idea documented nowhere.

PREFIX wdt: <https://portal.mardi4nfdi.de/prop/direct/>
PREFIX wd: <https://portal.mardi4nfdi.de/entity/>

SELECT ?item ?itemLabel
WHERE
{
  {
  SELECT ?item
  WHERE { ?item wdt:P31 wd:Q57162 .}
  ORDER by ?item
  LIMIT 5 offset 100000
  }
  SERVICE wikibase:label {bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en".}
}

[x] Generalize FHP Creation code to persons for 1000 persons cf. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MathSearch/+/954995
[x] fix MathSearch CI, cf https://phabricator.wikimedia.org/T345667
[x] Run the code 10 000 profile pages are created such as https://portal.mardi4nfdi.de/wiki/Person:Geir_Egil_%C3%98ien
[x] Generalize the code to deal with larger result sets https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MathSearch/+/955377
[x] Run code again,

it's running

subtask of https://github.com/MaRDI4NFDI/MaRDIRoadmap/issues/19

physikerwelt commented 10 months ago

@Daniel-Mietchen do you know by chance how to work around the timeout in SPARQL, I tried limit and offset, but the larger the offset gets the slower the query is.

Daniel-Mietchen commented 10 months ago

@physikerwelt

SPARQL optimization is generally query specific but here are some general ways to address the problem:

On the MaRDI endpoint, we can set the timeout limit ourselves.
For now, you can get a list of all humans in about 40s via SERVICE bd:sample { ?item wdt:P31 wd:Q57162 . bd:serviceParam bd:sample.limit 2000000 }, which is not SPARQL but a Blazegraph-compatible service that calls the Wikibase API. We could then have a non-SPARQL script loop over this list and query for additional information like labels.
We could call the API directly.
We could work with dumps and subsetting.

physikerwelt commented 10 months ago

All pages have been created, however, some pages have errors.

MaRDI4NFDI / portal-compose

Create person profiles #390