Tools for Profiling and Tuning Build Clusters

To make decisions about how to tune an icecc cluster, one needs real information about what happened during a distributed build scenario. We've set up a small icecc build cluster, are having trouble identifying bottlenecks, and are looking for suggestions for how to systematically collect build profiling information to help with the tuning effort. We currently have:

-vvv and -l args set in the daemons, and have access to the verbose console output from the machine initiating the build, as well as the iceccd logs from all machines.
real-time stats collected from the scheduler and provided by icemon and (my preferred) icecream-sundae monitors. -- I understand that restarting scheduler(s) will reset the stats that are displayed by these tools

While these do provide useful information for post-analysis and real-time monitoring, I'm looking for tools for measuring all of the details of a build scenario. We can post-process the console output to collect per-compile job information, but I'm also interested in gathering snapshots of the cluster state before, during and after a build scenario. We would use these snapshots as my barometer while tuning the system parameters. Looks like icemon and icecream-sundae don't have the ability to take "one shot" snapshots of the metrics that they're displaying.

I'm coming from distcc, which had a little-known feature that would serve build cluster statistics on a TCP port, by enabling these daemon options:

--stats
Turn on the statistics HTTP server. By default it is off. (Daemon mode only.)
--stats-port PORT
Set the TCP port to listen on for HTTP requests, rather than the default of 3633. (Daemon mode only.)

Is there any designed-in way to collect this info with these (or other) tools? I am now inspecting the source code of icemon and icecream-sundae to learn more about how they collect stats from schedulers and/or iceccd. Thanks.

icecc / icecream

Tools for Profiling and Tuning Build Clusters #513