Store batch run statistics on db

mozilla / spade

Automated scraping markup+CSS from a list of relevant URLs, using a variety of user-agent strings. Provides reporting on usage of CSS properties and apparent user-agent sniffing.

22 stars 9 forks source link

Store batch run statistics on db #63

Open maurodoglio opened 11 years ago

maurodoglio commented 11 years ago

We need to store the following stats for every batch run:

num pages scanned per minute (max,min,avg, std dev?)
memory usage (max,min,avg, std dev?)
number of responses per return code (20x, 30x, 40x, 50x)
number of cache keys hit/missed/stored
total bytes downloaded