Create competition status dashboard

[Tyler-Ward]

Create a dashboard or two to monitor the status of the competition services,infrastructure and metrics.

grafana might be a good candidate for this.

[PeterJCLaw]

Could you elaborate a bit on what sorts of information this would display.

While it's not exactly elegant, we do have https://github.com/PeterJCLaw/nwatchlive on the compbox configured to monitor various things, so I'd be interested to see what else we think we should display.

[Tyler-Ward]

Some ideas of thing's to display or record for later analysis. The main aim being to try and give more info to those on the tech deck. where as nwatch live returns bad/good these could also return more detailed info e.g. response times ect. Some of this data might also be good to have during the rest of the year as well like the stats you extracted from the IDE earlier,

Monitoring.

• Status of SR comp screens in other locations
• status of tech deck kit
• main Server status
  • CPU
  • memory
  • IDE usage
  • docs usage
  • response times

SRcomp status.

• Time delay in scoring matches.
• Percent of matches completed.
• more visualizations of league points e.g. histogram

Other

• Amount of live stream viewers

[PeterJCLaw]

Aha, it sounds like there's a bunch of different layers of things here, which are probably worth considering quite differently:

• infrastructure status (e.g: is the IDE up? are the compboxes in sync?); this is what nwatchlive currently aims to provide; failures at this level indicate a technical systems issue
• manual process performance (e.g: time delay in scoring matches); I'm guessing these are mostly useful for future improvements?
• teams dashboard (e.g: are the teams using the IDE?); I'm not actually sure how useful these would be, though I agree they might be interesting

• Status of SR comp screens in other locations

I'm not really sure what you mean by "status" here. Are you thinking of things like CPU usage, or more like having a mirror display showing what a given screen is showing?

• status of tech deck kit

Again I'm not sure what you mean by "status" here? Is this whether the kit is running or not? If so, then this strikes me (at least at our current scale) as having limited value -- I would expect that there will be a person at the tech deck almost all the time, so any issues would surely be caught by them anyway without the need for automated monitoring. Am I missing something here?

• main Server status
  • CPU
  • memory
  • response times

I'd not thought about monitoring these, though I agree that response times might be useful to gauge load. I suspect that we're probably going to just dial up the machine to be so ridiculously over-provisioned anyway that this won't be necessary (it's much cheaper to do that than it is to even bother instrumenting it IMO), though this does seem likely to be useful to add as a very low priority background project if someone has time.

• docs usage

This is somewhat harder to track directly as we're not hosting the docs. Google Analytics or piwik or something might work (though getting the data out of those services isn't particularly easy).

• more visualizations of league points e.g. histogram

This might actually be a fun thing to have available on the website too.

[Tyler-Ward]

@PeterJCLaw how hard would it be to get patience to dump the following data on a regular basis and post them as a web request at a server. total commits per team (can also be per project as long as we know what team owns the project). Accounts per team. last commit date per team.

[PeterJCLaw]

Not particularly, we have existing scripts which do much of what the latter two are asking for. Creating that sort of thing is probably a good first task if you want to get an understanding of the systems & puppet.

I thought this task was about stuff at the competition event though?

[Tyler-Ward]

The idea here is to track team development during the competition via number of commits, the others are as you look to already had scripts for them and might give us some insight into how teams work in the competition buildup.

[PeterJCLaw]

I really don't think this sort of thing is useful to track at the competition. Certainly the number of accounts per team is not going to be meaningfully changing at the competition.

If we want to do post-hoc analysis on this then I suggest that we do that using the actual commit data and server logs (nemesis logs all the interesting user account actions explicitly), at some point after the competition, rather than trying to create a proxy for the data during the competition.

[Tyler-Ward]

This has been created with some basic information, will add more dashboards/refine them more if i find more time. Login details are available from me if wanted.

[Tyler-Ward]

@PeterJCLaw is the monitoring module still enabled on the live server or was that removed in the rebuild at the start of the year?

[PeterJCLaw]

I think it's still there: https://github.com/PeterJCLaw/sr-server-puppet/blob/master/modules/sr_site/manifests/init.pp#L41-L44