search

tsafavi / codex

CoDEx: A set of knowledge graph Completion Datasets Extracted from Wikidata and Wikipedia
MIT License
153 stars 26 forks source link

Suggestion: Are individual files for entity extracts needed? #6

Open apoorvumang opened 3 years ago

apoorvumang commented 3 years ago

Hi, thanks for the dataset!

Unzipping extracts.zip for english took almost 30min on a RAID-6 HDD for me (since there are around 60k files). Wouldn't a single json be better, like entities.json? I have modified the code and create a PR (https://github.com/tsafavi/codex/pull/5), it might be useful for people who want a quick install on a server with slow disk writes.

tsafavi commented 3 years ago

Hey @apoorvumang, thanks for the good suggestion and apologies for my late response.

In your PR, can you also please update the quickstart section of the README to reflect that the user can choose between running the extract.sh script (individual files) or using the json file that you provide? Best to mention the tradeoffs between the two approaches -- disk write speed, memory usage, etc.

Also, ideally we would have entity_extracts.json for all languages, not just English. But I can handle that if it takes you too long to run the extraction for other languages.

Tara