Migrating old contents - Githubissues

simison commented 9 years ago

We'll need to migrate current wiki articles into a new format (semantic stuff) and merge hitchwiki maps into the wiki.

[x] Generate coordinate for each geolocation at the current wiki: countries, cities, airport, border crossings, highways, etc...
[x] Go trough all the city and country articles and wrap them into semantic wiki stuff
[x] Add semantic links between articles: connect cities to countries
[x] Import from old maps SQL all spots into separate articles
[x] Import from old maps SQL all ratings, waiting times and comments into new DB either directly via SQL or via the API
[x] Semantic-connect spots articles from # km diameter to city articles
[ ] Clean out <map lat='48.83398957668602' lng='2.34283447265625' zoom='10' view='0' float='right'/> tags
[ ] Clean out {{legend}}
[ ] Parse Infobox templates: Currently this stuff resides in Template:Infobox_Country:

{{Infobox Country
|in = 
|map =
|language = 
|capital = 
|pop = 
|currency = 
|hitch = 
|hitchbase = 
|avp = 
|links =
|paved = 
|BW =
}}

And this stuff in Template:Infobox_Location:

{{Infobox Location
|name = 
|name_native = 
|symbol = 
|country = 
|state = 
|state_symbol = 
|subdivision_type = 
|subdivision_name = 
|subdivision_symbol = 
|region = 
|region_symbol = 
|subdivision2_type = 
|subdivision2_name = 
|subdivision2_symbol = 
|map = 
|pop = 
|plate = 
|roads = 
|hitchbase = 
|links = 
}}

...most of it we can get for Geonames anyway, though.

omelnyk commented 9 years ago

https://gist.github.com/anonymous/478d83dddd6c780e9e34 Latest version. At this point every bug becomes a feature ;)

omelnyk commented 7 years ago

Necessary article migration script improvements for each processed article:

[x] clean out <map ... /> tags
[x] move text from the top of the article to Introduction field of the Semantic template
[x] (for cities and countries) fetch population from Geodata
[x] (for cities) fetch adminName1 from Geodata
[x] (for cities) extract plate and roads from old Infobox template
[ ] move links from old Infobox template to the bottom of article
[x] remove all location-related Infobox templates from the article
[x] (for countries) fetch CountryCode from GeoData
[x] (for countries) fetch Capital from oldInfobox Country template
[x] (for countries) fetch language from oldInfobox Country template
[x] (for countries) fetch currency from oldInfobox Country template (could be at GeoData, too)
[ ] (for countries) move avp from oldInfobox Country template to the bottom of article

See also #78 #121

omelnyk commented 7 years ago

Ideas to keep bot fail-proof:

[x] print out diff of article content upon saving
[ ] save changes in several goes (add SMW template, remove Infoboxes and <map ../>, ..)
[x] keep list of articles that have been migrated in a database table (similar to spotmigrate)

omelnyk commented 7 years ago

Idea: possibly use OrderedDict to preserve logical order of the fields in the generated templates.

omelnyk commented 7 years ago

Spot migration necessary improvements:

[x] need to set Country for each migrated spot (not trivial as our country names sometimes differ from Geodata's)
[x] implement country lookup in HWMaps-extension's HWFindNearbyCityApi

simison commented 7 years ago

not trivial as our country names sometimes differ from Geodata's

Perhaps that iso-code added to countries would help at that point, though?

I'd just keep it simple. Get countryname from the geodata -> check if article exists with that name -> add it to point. Otherwise leave empty? A few "USA/Russian federation" if/elses to capture most common weirdos.

omelnyk commented 7 years ago

To do (based on on last night's beta deployment issues):

[x] avoid country (and partially city) Geonames lookup throttling
[x] parse (227) infoboxes that contain nested templates
[x] skip continent articles (they contain weird markup that's making pywikibot time out)
[x] limit number of article save retries
[x] skip articles for which title retrieval fails and keep track of which these are
[x] use Unicode for country capital names
[x] allow for Area country lookup failures
[x] do export LC_ALL='en_US.utf8' before running bot
[x] interrupt migration if one step fails (set -e)
[x] add notice for each article that's blacklisted/whitelisted for semantic migration
[x] parse motorway templates nested in infoboxes
[x] run MediaWiki jobs between article, spot and extra migrations
[x] update folder permissions after extracting image tarball

omelnyk commented 7 years ago

[ ] Czechia --> Czech Republic for spot/page lookups

[ ] warning:

/var/www/scripts/bot/bot_install.sh: line 23: cd: .cache: No such file or directory
Cloning into '.cache'...
remote: Counting objects: 4625, done.
remote: Total 4625 (delta 0), reused 0 (delta 0), pack-reused 4625
Receiving objects: 100% (4625/4625), 5.73 MiB | 1.24 MiB/s, done.
Resolving deltas: 100% (4451/4451), done.
/var/www/scripts/bot/bot_install.sh: line 32: cd: pywikibot-core: No such file or directory
Cloning into 'pywikibot-core'...
remote: Counting objects: 38872, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 38872 (delta 2), reused 0 (delta 0), pack-reused 38861
Receiving objects: 100% (38872/38872), 32.85 MiB | 13.14 MiB/s, done.
Resolving deltas: 100% (29237/29237), done.
Submodule 'i18n' (https://gerrit.wikimedia.org/r/p/pywikibot/i18n.git) registered for path 'scripts/i18n'
Cloning into '/var/www/scripts/bot/pywikibot-core/scripts/i18n'...
Submodule path 'scripts/i18n': checked out 'cd138ef95047e2b4e304d934f640f90e1abf68a3'

[ ] add dummy save early on?

Password for user Hitchbot on hitchwiki:hitchwiki (no characters will be shown):
Logging in to hitchwiki:hitchwiki as Hitchbot
WARNING: API warning (login): Main-account login via action=login is deprecated and may stop working without warning. To continue login with action=login, see [[Special:BotPasswords]]. To safely continue using main-account login, see action=clientlogin.
WARNING: /var/www/scripts/bot/pywikibot-core/pywikibot/tools/__init__.py:1716: UserWarning: File /var/www/scripts/bot/pywikibot.lwp had 644 mode; converted to 600 mode.
Page [[15 (Denmark)]] saved

[ ] upgrade pip?

You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[ ] keep Category: entries out of introduction
[ ] keep __TOC__ and the like out of introduction
[ ] logging
[ ] look for "\n|" in spot descriptions and city/area/country introductions
[ ] use OrderedDict for semantic field values
[ ] see how migrate script handles border cities/spots
[ ] use bigger "spot belongs to a city" radius -- easier to unassign spots from a city, than to assign them
[ ] remove tables created by extensions we don't use anymore
[ ] graceful exit in spotmigrate and extramigrate if necessary tables don't exist
[] keep list of page titles per each type of error

[ ] don't add population if Geonames says it's 0, eg.:


#1513. Errachidia
http://beta.hitchwiki.org/en/Errachidia

Use {{City}} template

+++ @@ -1,10 +1,16 @@ -{{Infobox Location -|country = Morocco -|state = Meknès-Tafilalet -|pop = 95 265 -}} +{{City +|MajorRoads= +|Countries=Morocco +|Introduction=

''Errachidia''' is a very relaxed city in the southeast of [[Morocco]]. Many tourists pass by here on their way to Merzouga, but Errachidia itself is not very touristy. + +|LicencePlate= +|Location=31.9314,-4.42663 +|Population=0 +|AdministrativeDivision=Drâa-Tafilalet +|Bbox=-4.4768859,31.9077638,-4.3753052,31.9623573 +}}

== Hitchhiking out ==

Page [[Errachidia]] saved

omelnyk commented 7 years ago

aka "the neverending task" :)

simison commented 7 years ago

remove tables created by extensions we don't use anymore

We can do this manually.

Hitchwiki / hitchwiki

Migrating old contents #69