Stats generation needs improvement - backend processing

[GoogleCodeExporter]

This bug is about changing the _Snapshot entities to store better information.
Ref: 
https://code.google.com/p/swarming/source/browse/services/swarming/server/stats.
py#110

The problem at hand is that the stats framework is not calculating values 
properly. The issues falls in two buckets:
- The stats framework registers "events", not long term "tasks". So when a task 
is running, nothing is happening to note that the task is running. When a task 
completes, the total run time is registered at the minute where the task 
completed. That doesn't make much sense for a 1h task to have a single peak at 
the end of the task. So the values should be distributed across minutes.
- The stats framework currently only look at the logs but not at the actual 
entities. We should start looking at the entities, which stores most of the 
necessary information. This requires a significant revamping of main loop.

The net effect is that the value "Tasks active" is totally off the chart.

The code is at 
https://code.google.com/p/swarming/source/browse/services/swarming/server/stats.
py

Original issue reported on code.google.com by maruel@chromium.org on 6 Aug 2014 at 3:55

[GoogleCodeExporter]

Original comment by maruel@chromium.org on 6 Aug 2014 at 3:56

[GoogleCodeExporter]

Original comment by maruel@chromium.org on 6 Aug 2014 at 4:13

[GoogleCodeExporter]

Issue 102 has been merged into this issue.

Original comment by maruel@chromium.org on 6 Aug 2014 at 4:30