Mr0grog
commented
5 years ago First, a quick recap on how Scanner handles and imports data.
The web-monitoring-db project manages all the indexable metadata we have through a REST API. It doesn’t have any logic for interacting with external sources (e.g. Wayback, Versionista) itself — instead, new metadata for versions of pages comes in via a POST to /api/v0/imports. Separate scripts run as cron jobs and use APIs or scrape other services to load data, reformat it, and POST it to the imports endpoint. That metadata has to include a URL for where to retrieve the raw response body of the version and a SHA-256 hash of that data. If the URL is in an acceptable, publicly readable location, -db just stores the URL (so other users will access it later for diffing, etc). If not, -db downloads the content from the URL, verfies it against the hash, and stores it (there are multiple storage providers, but in production we use a publicly readable S3 bucket). (What URLs are acceptable is managed by configuration.)
The imports endpoint accepts data as JSON array or a newline-delimited JSON stream (using the application/x-json-stream content type). The stream is preferred because it’s more efficient to work with. Each post can have any number of versions, but to keep things manageable, we typically chunk them into no more than 1,000 entries per POST. You can also specify some options via a querystring (different sources have different shortcomings, and need slightly different treatment when importing). More docs at https://api.monitoring.envirodatagov.org/#/imports, though we haven’t document it super well. Improvements on those docs at edgi-govdata-archiving/web-monitoring-db#429.
The metadata for each import record generally needs to look like:
{
// URL of the page that was captured
"page_url": "https://www.hhs.gov/climate/",
// the time at which this capture was made (ISO 8601, W3C version)
"capture_time": "2018-10-08T14:56:36",
// URL of the raw capture this record represents
"uri": "http://web.archive.org/web/20181008145636id_/https://www.hhs.gov/climate/",
// hex-encoded SHA-256 hash of the capture's response body
"version_hash": "3fe44a54f9b77a12a12550921207d97b832309d6d840ebc5e62ff5f532cd3f12",
// The title of the page
"title": "Climate Change and Human Health | HHS.gov",
// optional list of individuals/orgs who maintain the page
"page_maintainers": ["NASA"],
// optional list of tags to add to the page (not the version)
"page_tags": ["site:NASA - science.nasa.gov"],
// identifies the source; it's an opaque string
"source_type": "internet_archive",
// any kind of useful additional metadata the source can provide
"source_metadata": {
"status_code": 200,
"mime_type": "text/html",
"encoding": "utf-8",
"headers": {
"content-length": "49139",
"x-varnish": "322830471 321326768",
"x-content-type-options": "nosniff",
"content-language": "en",
"server-timing": "cdn-cache; desc=HIT, edge; dur=44",
"strict-transport-security": "max-age=31536000;includeSubDomains;preload",
"x-varnish-cache": "HIT",
"server": "nginx/1.4.6 (Ubuntu)",
"content-security-policy": "frame-ancestors 'self' hhs.gov *.hhs.gov",
"connection": "close",
"link": "<https://www.hhs.gov/climate/index.html>; rel=\"canonical\",<https://www.hhs.gov/node/4916>; rel=\"shortlink\",<https://plus.google.com/+HHS>; rel=\"publisher\"",
"date": "Mon, 08 Oct 2018 16:39:13 GMT",
"x-frame-options": "SAMEORIGIN",
"x-content-security-policy": "frame-ancestors 'self' hhs.gov *.hhs.gov",
"x-akamai-transformed": "9 12254 0 pmb=mRUM,3",
"x-generator": "Drupal 7 (http://drupal.org)"
},
"view_url": "http://web.archive.org/web/20181008145636/https://www.hhs.gov/climate/",
"redirected_url": "https://www.hhs.gov/climate/index.html",
"redirects": [
"https://www.hhs.gov/climate/",
"https://www.hhs.gov/climate/index.html",
"https://www.hhs.gov/climate/index.html"
]
}
}The scripts we run as scheduled cron jobs are:
-
web-monitoring-versionista-scraperscrapes data out of Versionista, uploads it to S3, and formats and POSTs import records to -db. It started life as a relatively straightforward package for scraping Versionista (Versionista has no API), but has since gotten significantly more complicated and hairy. It also generates weekly task sheets for analysts, but that’s unrelated to imports. -
web-monitoring-processinghas a WIP PR (edgi-govdata-archiving/web-monitoring-processing#174) for the imports from the Internet Archive’s Wayback Machine. This also runs as cron-managed script.
Each of these extracts all the content it can find from a given timeframe in the specified source and formats it for importing as above, then POSTs it to the imports endpoint.
What would we need for Walk?
At the very least, Walk needs to provide a URL at which -db or an ETL script can retrieve raw response bodies, e.g:
GET /captures/{timestamp}/{url}Since -db really wants the body that would have ultimately been displayed in a browser from a given URL (i.e. after following redirects), I think Walk might want at least two URLs for getting bodies:
# The exact response body for a URL
GET /captures/raw/{timestamp}/{url}
# The *resolved* response body for a URL (that is, the body of the URL that was
# ultimately redirected to) -- this is the URL scanner cares more about.
GET /captures/resolved/{timestamp}/{url}In the most ideal case, a special handler/finalizer in Walk could do all the metadata formatting and POSTing to -db’s imports endpoint. If that were the case, we’d still need the above URLs, but no special ETL scripts and cron jobs sitting in the middle between Walk and Scanner.
Otherwise, those ETL scripts will need a way to get metadata for all the captures for a given set of domains or URLs over a given timeframe. I can think of a few ways to do that:
-
A really static system might have ATOM/RSS/JSON feeds of captures at predictable names and intervals, e.g. every hour:
GET /index/2018-11-01T00:00:00Z.json GET /index/2018-11-01T01:00:00Z.json GET /index/2018-11-01T02:00:00Z.jsonThose could contain all the metadata, too, or just pointers to URLs with the metadata.
-
A nicer system might support queries over timeframes:
GET /captures/meta?date=2018-11-01T00:00:00Z..2018-11-01T03:00:00ZThat would return a JSON array or newline-delimited JSON stream or maybe a CBOR array or stream of metadata for captures in that timeframe.
Additionally, we need a way to filter to only the URLs we care about:
-
If we have a Walk install dedicated to Scanner, this doesn’t matter (the only URLs it’s tracking are ones we care about).
-
If we had some way to name collections of URLs that Walk is scraping, we could query for "captures in this named collection."
-
Otherwise, this is more complex — 30,000 or 40,000 URLs is too many to ask for one-by-one, and probably too many to transmit as a query. For Wayback, we currently compose a list of domains and query the CDX API (which is basically an index) for all captures on those domains. On our side, we iterate through those and only request full data for the captures that might be ones we care about (unfortunately we can't do exact URL matches with Wayback, so we take the SURT key, make it even more generic [lower-case, no protocol, remove querystring], and compare that — it gets us a lot of false positives, but hopefully no false negatives). I think we’d do something similar for Walk.
One other bit worth noting in here is that Scanner uses SHA-256 hashes throughout. Ideally, Walk would just make those available (Wayback is a pain because they use base 32-enocded SHA-1, so we actually have to download the raw bodies and hash them ourselves).
Does that help?
Frijol
based on our Wednesday coordinating call! cc: @Mr0grog, @lightandluck, @Frijol, @jsnshrmn, @danielballan
Let's use this to conceive of walk as a service with an api that scanner uses to grab snapshots.