Logging/Monitoring for user services

idvoretskyi / horizon-cloud

An operations service to deploy, manage, and scale Horizon apps

0 stars 0 forks source link

Logging/Monitoring for user services #93

Open encryptio opened 8 years ago

encryptio commented 8 years ago

Seems to be complicated by the fact that kubernetes has a half-baked setup for this. See https://github.com/kubernetes/kubernetes/issues/24677, https://github.com/kubernetes/kubernetes/issues/23195, and https://github.com/kubernetes/kubernetes/issues/17183.

encryptio commented 8 years ago

Things to (maybe) include:

Textual:

hz serve logs
access logs (might not be possible due to the CDN in front)
rethinkdb logs
deploy log (list of user, time pushed, manifest size lines)

Graphs:

Number of active horizon sockets
RethinkDB system usage stats (CPU, memory)
RethinkDB internal usage stats (reads, writes per sec)

encryptio commented 8 years ago

I'm working on a pair of programs that will let us stream logs from the user pods back to a RethinkDB instance in the kube-system cluster. These can handle the hz serve logs and rethinkdb logs.

encryptio commented 8 years ago

Change of plans; we're still going to dump these into RethinkDB at the end for Horizon to query, but going to use kubernetes' existing fluentd to google cloud logging framework, pulling those into google pub/sub topics before they go to RethinkDB.

encryptio commented 8 years ago

Note to self: ensure an alert on the subscriber queue size has been added before closing