tom-james-watson
commented
2 years ago By using SPARQL, we can more easily create data sets for specific subsets of things like "Books", "Battles" or "TV Shows". We would still want an "All" collection though, which would work similarly to the current version.
To be more specific on what would be needed here, I'd need to explain a rough outline of how the current processing works.
- Take entire dump of english wikidata
- Loop over every entry, as long as the entry has one of the following date values - https://github.com/tom-james-watson/wikitrivia-scraper/blob/0858260/src/main.rs#L32-L48
- Process each item. The main chunk of that is happening here: https://github.com/tom-james-watson/wikitrivia-scraper/blob/0858260/src/item/process.rs#L266-L271. Here we reject items based on a few checks like undesirable
instanceofs, labels/descriptions that are problematic etc. - We then also need to only include items that are sufficiently well-known. This is the difficult part. One vector for checking this is having a minimum number of wikidata sitelinks, which acts as a first filter as it is included directly in the dumps. We then also make a request to the wikipedia page views API to check how many views the associated wikipedia article gets, and also filter based on that.
In order to use SPARQL, we would need to be able to not only get enough items to populate the game, but we also need to be able to filter based on some heuristics to ensure that we only generate interesting cards that can reasonably be answered. A great vector for that is wikipedia page views, but there may also be better ways of doing this.
The current game has a list of around 10,000 cards, to give an idea of how many would be needed.
It's worth noting also that the kinds of cards that could be generated could be improved a lot as well. For example, you may have a card for Woodrow Wilson, who is well known enough to be included, but the card would be for when he is born, which is much more difficult. A better card would be to ask when he became president. Being able to detect what it is that is interesting about an entry and programmatically generating a card based on that would be great, though I imagine difficult.
What I get the feeling may make more sense would be to have many different SPARQL queries that specifically compose data based on things like the above example, so in that case a list of when famous world leaders came to power, and then to join all those datasets together.
It would also be possible to pass the results of a SPARQL query through another processing step to check the wikipedia pageviews should we feel like the wikidata sitelinks still doesn't provide a good enough vector for how well known the item is.
Some initial queries I was playing around with:
namedgraph
kidehen
tuukka
Theo-Strongin
I had trouble when I tried using SPARQL, which was actually my first approach to tackling this. Any queries that would be broad enough and return enough data would just time out. Maybe people in the community with more experience with SPARQL can help though!