Wikis that have dumps (e.g. wikimedia ones)

blester125 commented 10 months ago

Several datasets some from Wikimedia sources and they provide data dumps. These dumps contain wikitext markup which can be parsed with libraries like wtf_wikipedia.

[ ] Wikipedias (Encyclopedia)
- ~10m pages
- ~100 languages
- Moderated
- meta-dump includes "talk" pages
[ ] Wikinews (News)
- ~10k pages
- ~10 languages
- Moderated
[ ] Wikibooks (Textbooks)
- ~100k pages
- ~10 languages
- Moderated
[ ] Wikitravel/Wikivoyage (Travel guides)
- ~100k pages
- ~10 languages
- Moderated
[ ] Wikisource (old stuff)
- ~100k pages
- ~10 languages
- Moderated
[ ] Wikiversity (Learning Resources/Textbooks)
- ~100k pages
- ~ 10 languages
- Moderated
[ ] Wikipedia Talk Pages (Dialog-ish)
- ~1m pages?
- ~100 languages?
- Not Moderated
- Some hate speech/noise

craffel commented 9 months ago

@conceptofmind was also asking me about mediawiki parsing, so maybe coordinate to make sure you aren't overlapping work. The main thing we discussed was that I suggested using ncermountain/wtf_wikipedia plus a little bit of postprocessing.

conceptofmind commented 9 months ago

@conceptofmind was also asking me about mediawiki parsing, so maybe coordinate to make sure you aren't overlapping work. The main thing we discussed was that I suggested using ncermountain/wtf_wikipedia plus a little bit of postprocessing.

Will talk to Shayne and Luca to ensure that there is no overlap on this.

shayne-longpre commented 8 months ago

https://github.com/spencermountain/wtf_wikipedia

conceptofmind commented 7 months ago

https://github.com/spencermountain/dumpster-dip/issues/4

blester125 commented 3 months ago

Subsumed into https://github.com/r-three/common-pile/issues/82

r-three / common-pile

Wikis that have dumps (e.g. wikimedia ones) #1