Switch to structured logging

[yuvipanda]

Either with logfmt, structlog or just raw JSON logging.

Partially for doing https://github.com/jupyterhub/mybinder.org-deploy/issues/97 in a way that's agnostic to the cloud provider we run on, but also because it makes searching through logs, etc easier

[betatim]

I need a bit of help here. I can easily imagine bhub creating JSON formatted messages. Where would it send them? Is logstash the right thing to learn about? In my mind we are looking for something that is a glorified FIFO buffer of ~100MB or so. bhub sends its JSON formatted messages there, the buffer buffers them, a web frontend lets users stream (and go back in history) new incoming messages.

As you can tell from the above: I have no idea where to get started in terms of existing tools for this. However I want to have a webpage I can put on a big screen to see the heartbeat of binder. So if someone could enable me that would be great :)

[yuvipanda]

We already do this in repo2docker with https://github.com/madzak/python-json-logger. We just hook into python logging and produce log messages that get sent out to stdout in JSON format. We then tail those logs somehow (in our case, reading with kubectl logs or from calling the stackdriver API), and extract the info from there. This is the easiest way to do this that's also not cloud-provider specific nor affects people who don't want to have a public feed.

[choldgraf]

Just a note that we've got some other logs discussions here #75 #155

Would structured logging be a problem to solve separately from persistent logs? Or should they be solved simultaneously?

[yuvipanda]

@choldgraf it's unrelated. Structured logging would just help with debugging binderhub itself, and is unrelated to storing user build logs.