make_wikipedia.py hardcoded to simple

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

https://allenai.github.io/dolma/

Apache License 2.0

976 stars 108 forks source link

make_wikipedia.py hardcoded to simple #57

Closed peterbjorgensen closed 1 year ago

peterbjorgensen commented 1 year ago

How to fix: Change the URL from: https://dumps.wikimedia.org/simplewiki/{date}/{lang}wiki-{date}-pages-articles-multistream.xml.bz2 to: https://dumps.wikimedia.org/{lang}wiki/{date}/{lang}wiki-{date}-pages-articles-multistream.xml.bz2