blester125
commented
3 months ago The wikiarchive wikis are chugging along, I think the main bottleneck is I/O contention so there isn't much we can do to speed it up.
The wikiarchive wikis don't tend to include the talk pages, which is another reason we will want to re-scrape in v2
The wikimedia data I have processed (wikipedia, wiktionary, etc) didn't have talk pages either. I'm updating code, redownloading, and reprocessing to include the talk pages (will post some stats about how much more data that is when done)
wikitext parsing updates
wtf_wikipedia seems to work well. One place it leaves weird artifacts is images with really long (could be caused by newlines) captions, it leaves the [[ in there. It also remove section names, but I have an easy fix to get them back.
I've also been working on handling math parsing. wtf_wikipedia is pretty inconsistent on how it handles maths. It handles simple stuff well but strips out anything more complex. When processing 1 shard of the wikipedia dataset, 49% of the <math> tags got stripped out (of those, 91% of the more complex <math display="block"> sections were stripped). It does a bit better with the math template syntax that wikitext supports (but that might be because people tend to use it for less complex stuff). Only ~5% of those are totally stripped, but it's processing of the ones that aren't removed is still wrong, the clearest way is that it removes the <sup> and <sub> tags.
My current approach is to:
1) convert all the <math> tags to appropriate $ and $$. This converts it to latex and seems to cause wtf_wikipedia to leave it alone
2) detect the {{math|...}} templates in the wikitext and either:
a) remove them from the wikitext, leaving a simple marker, convert the template into latex, use wtf_wikipedia to convert the rest of it, and substitute them back in at the correct place. An issue with this is that the templates can contain random wikitext like the link syntax so we'll need to ~reimplement some wikitext parsing into math template -> latex conversion code.
b) Do rewrites in the templates for things like <sub>/<sup> into a more latex-like format and then pass things into wtf_wikipedia. It it unclear how many of these rewrites will be needed to avoid stripping these templates (for example wtf_wikipedia can handle a template like {{math|1=F=ma}} but will strip a template like {{math|F {{=}} ma}}. We would also want to add $ around these parts.
Some mathlike templates (for example {{mvar|X}} can appear without the {{math|...}} template so my plan is to find the {{math|...}} ones first, which will contain most of the other template, and then pick up the other ones later.
An open question is the handling of unicode symbols. Lots of math articles have symbols like π or θ directly. In a latex conversion would we want to convert these to \pi and \theta? Similarly, there are fraction templates like {{sfrac|1|2}} that wtf_wikipedia converts to 1/2 but might make more sense to have as \frac{1}{2}
I might convert all the dumps with just wtf_wikipedia as a v0 it get approximate token counts while I work on refining the math handling.
The end of the wikipedia page is generally a long list of references, links to related pages, etc. wtf_wikipedia converts these in a more plaintext format, but it is still pretty long and unnatural. I'm thinking of looking into removing these final sections.
craffel
spencermountain
This issues tracks all wiki processing as the different sources have been unified. Will be closing https://github.com/r-three/common-pile/issues/7 and https://github.com/r-three/common-pile/issues/1
There are three sources of wiki data we will be using:
Sources to create a list of wikis:
Data Processing Pipeline:
Data Collection
Archive Sources:
wikiteamcollection. When you allow thewikicollectionstag too it jumps to ~4million wikis.--worker_idand--num_workersmetadata.identifierfield is used as the key, but our actual "source" field will use the domain name.wiki/archivewiki/dumpwiki/scrape${target_dir}/${id}to${target_dir}/${id}[:n]/${id}[n:2n]/.../${id}. Things likeos.path.exists`, used to check for data already processed are currently really slow (~40 minutes to check if all ~325,000 exist on disk), this should help speed things up."text"field has wikitext in it.Dump Sources:
"text"field has wikitext in it.Currently unclear if a dump grabbed based on IA metadata should be saved into the same area as the wikimedia dumps where processing is based on what lives in the dirs or it should stay in the IA area where the metadata file dictates what is processed.
Scrape Sources:
wikicollectiontag in addition to thewikiteamtagWikiText parsing:
Now that everything is in a unified sharded format, actual parsing of wikitext is essentially infinitely horizontally scalable.
"text"field to plain text.