WardCunningham
commented
5 years ago I've revised the json.rb converter to record in the output more information about decisions it has made. This lets me assess results by selecting subsets with jq. Here, for example, is a successful recovery:
{
"date": "November 3, 2011",
"text": "We aim to make simple things simple and complex things possible...",
"rev": "22",
"page": "AlanKayOnSmalltalk",
"copy": true
}On complete failure I still produce the page name so that I can apply more detailed diagnostics driven from jq results.
{
"page": "AdelinoRodrigues",
"trouble": true
}See where I begin diagnosis with jq -r 'select(.trouble)|.page' above.
btrower
We add some tools for finding and correcting character code problems. Rather than work on the whole wiki database, we select out troublesome pages into a
troubledirectory. As we improve our json.* script chain we will produce possibly improved pages in apagesdirectory. From this we can select out specific remaining problems which we diagnose at the line level and collect in theinvaliddirectory. From this we inspect the octal characters and find substitutions then repeat the process looking for improvements.bulk convert
This reports progress on individual files using the codes
.,,andxfor retrieved text, copy or still trouble. Counts will measure progress.diagnosis
A typical invalid file shows numbered lines with invalid characters.
Good technique is to flip through the reports working from the shortest entries first.
When a particular line of a particular file is of interest, isolate that line and dumb it in octal.
In this case the invalid character is octal 231.
substitution
Perl is happy to change these characters to something preferable. In this case, the
tmis probably a joke and won't be missed.These commands go into
json.plandcheck.plwhich have a similar structure. We repeat from the beginning and find less work to do. (In this case, way less work todo having worked a few substitutions before repeating.)