Caching sphere crawler

[cdata]

When we eventually implement proper multiplayer, we will need to take every opportunity we can to minimized perceived latency as clients traverse the Noosphere. Immutable data and content addresses afford us the ability to constantly, preemptively cache content that we can resolve the addresses for. This would not be practical for a mobile client to do, but it is the perfect job for the gateway. Once cached, these blocks are only one network hop away from the client.

Design notes

• We can seed the crawler with the spheres that are named within a given address book
  • These spheres are highest priority because they are the spheres that are easiest for the user to link to
  • By adding a name to the address book, the user "magnetizes" their sphere to the blocks of another sphere, pulling them into gateway locality
• The crawler can be configured to cache N-deep spheres, in which case it can recurse through the subsequent spheres' address books (or until some soft block limit is hit)
• The pet name system (#12) should feed a work queue that is consumed by the crawler so that it picks up new revisions as they are published to the DHT

[cdata]

We could call it "Orb Weaver," cause it's a spider (in the web tradition) that crawls spheres.... :nerd_face: