Provide the option of tracking crawling statistics

[nietaki]

Implementation detail ideas:

• off by default
• separate API function for it, like crawl_with_stats(source, parser_logic, options \\ []) :: {flow, ref}
• create a :simple_one_for_one StatsSupervisor as part of Crawlie app
• use the ref (a {:stats, PID} tuple?) to send the stats updates from the Flow and for retrieving the stats from the Supervisor children.

[nietaki]

Working on it in https://github.com/nietaki/crawlie/tree/stats

[nietaki]

Stats to track (?)

• [x] pages visited counter
• [x] links extracted counter (including duplicates)
• [x] retry distribution
• [x] total bytes received (response bodies only 😞 ) (counter)
• [x] status codes distribution
• [x] simple content-type distribution?
• [x] failed fetch pages urls (set)
• [x] failed parse pages urls (set)
• [x] status: :ready | :working | :finished
• [x] total time spent (start time and end time)
• [x] cumulative time spent waiting on responses (counter)