Mirroring a wiki

[lahwaacz]

References:

• https://github.com/WikiTeam/wikiteam/
• https://www.mediawiki.org/wiki/Manual:Grabbers
• https://www.mediawiki.org/wiki/API:Database_field_and_API_property_associations
• https://www.mediawiki.org/wiki/Incremental_dumps
• https://dumps.wikimedia.org/other/incr/

To do list:

• [x] abstraction for working with an SQL database
  • only a couple of tables of interest, similar but not exactly the same schema as MediaWiki has (compatibility only via the API layer - see below)
  • for inspiration:
  • https://github.com/rkabir/mwdb
  • http://pythonhosted.org/mediawiki-utilities/core/database.html
  • https://github.com/mediawiki-utilities/python-mwdb
• [x] grabbers for fetching important tables from API
  • [x] namespace, namespace_name (custom tables)
  • [x] recentchanges, logging
  • [x] user, user_groups, ipblocks
  • [x] page, page_props, page_restrictions, protected_titles
  • [x] archive, revision, text: wait for list=allrevisions module in MediaWiki 1.27 https://phabricator.wikimedia.org/T113885
  • [x] tags
  • [x] interwiki
• [x] handle difficult actions involving DELETE or UPDATE queries as part of the syncing process:
  • [x] removing from user groups
  • [x] unblock
  • [x] unprotect
  • [x] delete (move from revision to archive)
  • [x] undelete (move from archive to revision, also check page_id)
  • [ ] selective undelete
  • [x] merge (works assuming that both source and target page were not deleted before the sync)
  • [x] import (works assuming that the imported pages were not deleted/merged/whatever before the sync)
  • [x] delete/revision, delete/event
  • [x] tag/update (separately for recentchanges, logging, revision, archive)
  • other log events: https://wiki.archlinux.org/api.php?action=help&modules=query%2Blogevents
• [x] let the SQL database serve as a source of data instead of the API
  • [x] list=recentchanges
  • [x] list=logevents
  • [x] list=allpages
  • [x] list=protectedtitles
  • [x] list=allrevisions
  • [x] list=alldeletedrevisions
  • [x] titles=, pageids= for use with prop=
  • [x] common executor for the DB select queries (for easy profiling)
• [x] framework for tests
  • [x] pytest fixture for web server (nginx)
  • [x] pytest fixture for php-fpm
  • [x] pytest fixture for MediaWiki installation (depends on nginx, php-fpm, postgresql + MW sources, config, initial SQL)
  • [x] write the tests...
  • implement a double-source wrapper, which yields from the API and checks the DB selects, ignoring NotImplementedErrors etc. (usable for unit tests as well as real-world testing) split into #50

[lahwaacz]

Upstream bug reports which block further improvement:

• archiving:
  • [ ] list=alldeletedrevisions returns pageid=0 instead of ar_page_id
  • [ ] Make ar_parent_id visible through the API
  • [x] Use ar_parent_id on undeletion
• merging:
  • [ ] MergeHistory: record the target page ID in the log events (merged into log_params doesn't contain the page ID of the target page in merge events)
  • [ ] MergeHistory: timestamp-based midpoints are not sufficient

[lahwaacz]

This is somewhat finished and working nicely, so it's time to close this.