Wikimit is an in-progress Wikipedia-to-git converter. This document details the approach.
Goals
Input: wikipedia article, output: github repo organized under wikimit-hub
Anonymous access (no account needed)
Requesting conversion should be idempotently create/update a repo
Low costs by using as-needed hosting and free storage
Architecture
Overview
Sequence (a):
User chooses a Wikipedia page
User submits URL in wikimit site
Site sends ajax request
Request handler adds URL to queue and returns the expected github URL
Site polls github and shows link to user when ready
Sequence (b):
Sync agent pulls URL off queue
Sync agent queries Wikipedia for revision history
Sync agent pushes commits to GitHub
wikimit.org
The wikimit site is a static webpage, hosted on GitHub pages with a custom domain (wikimit.org). The page has a textbox and a submit button.
Clicking the submit button sends an ajax request to the backend handler, which responds with a github URL. The client-side JS then polls that URL and alerts the user of two developments:
When the repo is created
When the repo is up to date with the latest revision
When (2) is reached, polling stops.
Request handler
The request handler is implemented as an AWS lambda. It validates the incoming Wikipedia URL, converts it to a GitHub repo URL, and places the pair on the job queue.
The handler will also:
Make sure that the URL isn't already on the queue before putting it on
Fail if the queue is full
Throttle by IP address
Job queue
The job queue is a short-lived list of URLs that need to be processed.
TBD on the hosting.
Sync agent
The sync agent monitors the job queue and is only active when there's work to be done. Jobs can be done in parallel, but must take a lock on the queue when modifying it.
To do a job, the agent first creates a GitHub repo if one doesn't already exist. Otherwise, it clones the existing repo to its local filesystem. Then it fetches some number of revisions from Wikipedia, commits them, and pushes to GitHub. Then it (a) removes the job it just did from the queue, and (b) pushes a follow-up job to the end of the queue (if needed).
Follow-up jobs are needed if:
There are more revisions to process.
The job failed. In this case, the job is marked as being a retry.
TBD on the hosting.
Considerations
Security
Only the request handler and sync agent have access to the job queue (TBD on details).
The request handler throttles by IP (TBD on details).
The sync agent has access to a GitHub account with limited permissions, such that it can only push to the wikimit-hub project.
Reliability
Pages that have a huge number of revisions don't prevent other work from being processed because of the "do a small amount of work and create a follow-up job" approach the sync agent takes.
Transient failures are dealt with by retry jobs, and if they're persistent the job is dropped.
Alerting on:
Queue is full
Dropped jobs
Legal
Wikipedia content can be redistributed as long as it uses the CC-BY-SA license.
Wikipedia limits access to their API at two simultaneous requests per IP.
Performance
Initial testing shows that converting a page with 1000 revisions takes about 100 seconds, with 90% of time running "git commit". This is pretty slow but not catastrophic.
A small article with 1000 revisions takes about 6 MB of disk space. This should be manageable.
Both speed and size need to be stress tested with larger articles. TBD.
Design doc for wikimit
Intro
Wikimit is an in-progress Wikipedia-to-git converter. This document details the approach.
Goals
Architecture
Overview
Sequence (a):
Sequence (b):
wikimit.org
The wikimit site is a static webpage, hosted on GitHub pages with a custom domain (wikimit.org). The page has a textbox and a submit button.
Clicking the submit button sends an ajax request to the backend handler, which responds with a github URL. The client-side JS then polls that URL and alerts the user of two developments:
When (2) is reached, polling stops.
Request handler
The request handler is implemented as an AWS lambda. It validates the incoming Wikipedia URL, converts it to a GitHub repo URL, and places the pair on the job queue.
The handler will also:
Job queue
The job queue is a short-lived list of URLs that need to be processed.
TBD on the hosting.
Sync agent
The sync agent monitors the job queue and is only active when there's work to be done. Jobs can be done in parallel, but must take a lock on the queue when modifying it.
To do a job, the agent first creates a GitHub repo if one doesn't already exist. Otherwise, it clones the existing repo to its local filesystem. Then it fetches some number of revisions from Wikipedia, commits them, and pushes to GitHub. Then it (a) removes the job it just did from the queue, and (b) pushes a follow-up job to the end of the queue (if needed).
Follow-up jobs are needed if:
TBD on the hosting.
Considerations
Security
Only the request handler and sync agent have access to the job queue (TBD on details).
The request handler throttles by IP (TBD on details).
The sync agent has access to a GitHub account with limited permissions, such that it can only push to the wikimit-hub project.
Reliability
Pages that have a huge number of revisions don't prevent other work from being processed because of the "do a small amount of work and create a follow-up job" approach the sync agent takes.
Transient failures are dealt with by retry jobs, and if they're persistent the job is dropped.
Alerting on:
Legal
Wikipedia content can be redistributed as long as it uses the CC-BY-SA license.
Wikipedia limits access to their API at two simultaneous requests per IP.
Performance
Initial testing shows that converting a page with 1000 revisions takes about 100 seconds, with 90% of time running "git commit". This is pretty slow but not catastrophic.
A small article with 1000 revisions takes about 6 MB of disk space. This should be manageable.
Both speed and size need to be stress tested with larger articles. TBD.
Costs
Hosting solutions:
TBD on actual cost estimates.
Open questions
See all the TBD items above.