Support incremental redownload of sites with new page versions

[davidfstr]

Since Crystal's inception it's been a long-term design goal to just not just allow websites to be downloaded once, but to support being downloaded multiple times at different points in time.

For example: It should be possible for me to download the entire xkcd site (with all comics) today, and then next week efficiently download the new set of comics that have appeared since then.

Designing an easy-to-use workflow to support this scenario is not trivial. Many challenges are involved.

Related Tasks:

• [x] Support multiple revisions of a URL in the data model (with multiple ResourceRevisions per Resource)
• Manual workflow v1†:
  • [x] It is possible to define a --stale-before date(time) such that all resources downloaded before that date(time) are considered stale
  • [x] Pressing Download on a "downloaded but stale" member will redownload it and mark it as "downloaded and fresh"
• Manual workflow v2:
  • [ ] "Hard Refresh": Can manually force redownload of an URL (and any embedded resources) in the GUI (ex: "Home" for xkcd).
    • This will discover links to new members of existing URL groups (ex: "Comics" for xkcd)
  • [ ] Can see in GUI that an existing URL group does contain new undownloaded members
    • [x] It must be possible to see the different statuses of an URL in the UI: "undownloaded", "downloaded and fresh", "downloaded but stale" (#35)
  • [x] Can manually redownload a group, such that its newly-discovered undownloaded members are downloaded
• Semi-automated workflow:
  • [ ] Can mark URL (group) as "considered stale after X time passes"
  • [ ] Can see in GUI that an existing URL (group) is or does contain new "downloaded but stale" members (#35)
    • At this point, opening a project should reveal that some URL (groups) are now immediately stale and in need of redownload
  • [x] Pressing Download on a "downloaded but stale" member will redownload it and mark it as "downloaded and fresh"
• One-click automated workflow:
  • [ ] After opening a project, there exists a button (or similar) the user can press to automatically redownload everything that is stale
• Zero-click automated workflow:
  • [ ] It is possible to deploy Crystal as an always-running service, with a set of (cloud) projects it does keep fresh automatically

---

† "Manual workflow v1" is problematic:

• It requires hard-switching a project into a "redownload everything" mode. Clumsy. Heavy-handed.
• It requires site-specific knowledge (of individual URLs that are expected to expire).
• It requires manually tracking which new URL group members were discovered (perhaps via shell inspection) so that only the "undownloaded" are manually individually downloaded. Error prone. Labor intensive.
• It relies on server-provided HTTP Date + Age headers to actually be correct, when there is no guarantee they will be. (See #89 for a longer explanation.)

Therefore "Manual workflow v2" has been introduced as an alternative instead.

[davidfstr]

Site-provided staleness/freshness signals

ETag & Last-Modified Challenges

In theory sites may advertise for each page - using the ETag and Last-Modified HTTP headers - whether it has updated content vs. when it was last fetched by a browser. But there are many issues:

• Very many dynamically generated sites do not generate an ETag or Last-Modified HTTP header at all.
• Many sites choose to generate a "strong ETag" which looks for byte-for-byte equivalence rather than a "weak ETag" which looks for content equivalance. Crystal cares about the latter more than the former for efficient re-crawling.
• Even if a weak ETag is generated, the site's definition of the "content section of the page" which has changed may differ from the definition preferred by the operator of Crystal, causing unpredictable recrawls and non-recrawls (if the ETag were to be trusted).

For example the xkcd site - which is good enough to generate an ETag at all - makes an ETag that looks like "62e1f036-1edc":

• The part before the dash is a hash of the entire page.
• The part after the dash is a hash of the content area of the page.
• Together, the entire ETag is a "strong ETag".

Sitemap Challenges

Sites may have a sitemap.xml that they generate for search engines. These sitemaps do provide a per-page "last modified" date, but it is common for a sitemap to set ALL pages to the same latest date that the entire site was regenerated.

RSS/Atom Challenges

Sites may have an RSS/Atom feed that references post-like pages on the site. The "post date" given for a particular page usually is a stable value that maps to when a post-like page was created. However that date is frequently not updated if an existing post-like page's content is modified.