Retrieving the identifiers by occupation/instance of should be done against Wikidata dump - Githubissues

Wikidata / soweego

Link Wikidata items to large catalogs

https://meta.wikimedia.org/wiki/Grants:Project/Hjfocs/soweego_2

GNU General Public License v3.0

97 stars 9 forks source link

Retrieving the identifiers by occupation/instance of should be done against Wikidata dump #138

Open MaxFrax opened 5 years ago

MaxFrax commented 5 years ago

IMPORTANT: do not use this method if paged SPARQL queries work fine.

~~We are waiting for direct access to the Wikidata dumps in the VPS machine:~~ ~~https://phabricator.wikimedia.org/T209818~~

marfox commented 5 years ago

Workflow example: MusicBrainz musicians. INPUT: NT (triples) dump, i.e., wikidatawiki/entities/latest-truthy.nt.bz2;

[ ] get all sub-classes of musician Q639669;
[ ] for each sub-class + musician, filter all subjects (QIDs) with predicate occupation (P106) and object sub-class;
[ ] for each filtered subject, if there is a triple with predicate MusicBrainz ID (P434), remove it from the set.

marfox commented 5 years ago

We are waiting for direct access to the Wikidata dumps in the VPS machine: https://phabricator.wikimedia.org/T209818

Task resolved: ls /public/dumps/public/wikidatawiki/entities

marfox commented 5 years ago

Alternative SPARQL method discussed during WikiCite 2018: unwind subclass of recursion. See https://etherpad.wikimedia.org/p/WikiCite18Day3sparql

marfox commented 5 years ago

One-shot BASH done

marfox commented 5 years ago

We finally opted for paged SPARQL, leaving this open as an extra feature.