A integration tests should periodically run to inform health of entire crawl

currently, when users download neume-network/core and run it, there's a high chance that bugs in the crawl prevent it from concluding
generally, although we have many unit tests, the integration between modules can lead to errors that we're currently not tracking. The more we test, the more apparent these errors will become and we can increase the turn-around time to fix them
hence, there should be an integration tests that runs periodically, e.g. once a day, that downloads the entire neume-network/core codebase, installs all dependencies and starts running a real crawl
finally, after the crawl has finished, some tolerant health measures should be evaluate (e.g. if all data-dir files of all existing strategies have been created)
the health measurement shouldn't enforce strict assertions e.g. (assert(file.length, 1234) as we must be conscious of the non-deterministic nature of the crawl

neume-network / core