Client-Side Scraping for Site Distillation

[cretz]

• Imagine if sites you read a lot were prescraped and given to you in simple markup like HN or Craigslist

• Should be read only and have links to full sites

• Probably just a local webserver in the background

  • Actually, if this were a native desktop app, it would provide many more benefits
  • But could not do on iOS because they wouldn't allow scraping script downloads or scraping in general against other site's ToS
  • Maybe always on at home and visible via onion service or similar on mobile

• Offers more tracking protection

• Login is interactive and cookies are stored but will prompt user when you have to log in again

• Not trying to aggregate or anything here, each site is its own thing still

• All scraping is done client side

• Works off a curated set of scraping scripts that can be updated quickly to stay ahead of sites trying to subvert

• Can also support things like notifications on freshly scraped items

[cretz]

Very early stages at https://github.com/cretz/ownweb

[cretz]

Now probably handling this at https://github.com/cretz/recapitate, node is just an easier platform for this than Go