Lightweight client library; monitoring of multiple apps

[thoughtpolice]

So, I really like ekg right now, but one thing I've been wondering about for the past few days is if it would be possible to start an EKG server, with arbitrary remote clients being able to connect back to it and display stats.

My main reasoning for this is that 1) it could significantly reduce the dependency chain needed in a project to use ekg (snap pulls in a lot of extra stuff.) It could be reduced significantly with a change like this. I always tend to be somewhat wary about such huge chains, although most of the time it works fine. There may be a lighter weight HTTP library that would work and alleviate this, but my other reasoning 2) is that eventually it would be nice if you could have an EKG server, with multiple remote clients connected, and the ability to get an overview of all the connected clients - think a Cloud Haskell application, all reporting GC statistics/other stuff to one central monitor server. Even withstanding cloud haskell, this could be very useful if you have multiple Haskell services doing different things as different processes, even on different machines - web server, caching layer, daemon, other monitoring apps, etc.

So for example, there could be the regular ekg package which supports starting servers for a specific application via System.Remote.Monitoring like right now, but the package could also provide, say a simple 'ekg-server' executable you can start. Client applications could then depend on a ekg-client library of some sort, with a similar interface like the one provided now, and specify a remote server to report statistics to.

Thinking about it, to avoid the dependency chain, the current code would need to be stratified a bit; for representing say, the core JSON types, and the communication protocol, apart from the networking components, and the server. So there would need to be packages like:

• ekg-types or ekg-common, which defines the JSON types and protocols used for communication on all fronts.
• ekg which depends on -common/-types, as well as snap, that looks pretty close to how it does now. It offers the server component that you can embed directly into an application right now, and also offers a 'master server' executable which can handle multiple clients.
• ekg-client which depends on -types/-common, and offers simplistic network functionality that reports GC stats to a remote server.

I realize that having packages like this which must be worked on in lockstep is somewhat troublesome, but it does help in using pieces in isolation (and it seems to work pretty well for say, Yesod, which is a much huger project with many more intermittent packages.)

There are a lot of important other things to take into account for a 'full featured' system, such as aggregate views, and authentication of clients. But I think that just having a server that can handle multiple clients, and splitting off functionality into a client library is a good place to start and not insurmountable by any means. Ad-hoc and one-off applications can always use the simplistic module in the ekg package, but having multi-app monitoring could be incredibly useful once you begin having multiple components that operate as separate processes/on different systems.

Johan, are you opposed to these ideas? If not, I may try working on this a bit in the near future and reporting back here. I would like to see ekg become a great library for live performance monitoring (an area that is severely lacking in solutions,) and multi-system monitoring would be a great feature to have, but I do understand if the ideas above don't quite fit into your view of what abstractions you want to provide.

[tibbe]

Here's what I've planned for ekg at the moment:

1. Pull out the Counter and Gauge types, plus the internal type used to store registered counters, into a new package. This would let libraries, in particular low-level libraries, to export counters without depending on snap-core, snap-server, or any of their dependencies.
2. Pull out the protocol parts (e.g. the ToJSON instances) into a separate package to allow HTTP frontends to be written in any of the major web frameworks. This would allow people to embedded the monitoring functionality in a bigger webapp (e.g. as snaplets or Yesod subsites.)

I'd also like to use a much more lightweight HTTP server, unfortunately we don't have one that's both has fewer dependencies than Snap while being convenient enough to use (i.e. provide HTTP header parsing and so forth.)

I don't intend to add multi-process data collection to ekg. What I intend for people to do is to build meta-monitoring applications that

• pull data from several ekg-enabled processes, using the JSON API,
• aggregate the data over a longer time period, and
• serve that data over HTTP using some fancy UI that e.g. allows users to dig down into historical data.

In addition such meta-monitoring apps could set off alarms automatically if certain monitored variables went above/below some given thresholds.

Finally, I intend for ekg to be pull based rather than push based. However, after the split into 3 (or so) packages it should be possible to write a push based client that uses the same protocol and counter store.

[gseitz]

I stumbled upon ekg on @ocharles' blog and started stripping ekg down to ekg-common in my fork.

• I removed all Snap related code and introduced Registry as a way to bundle together various metrics.
• I also added a pull-based metric called PullGauge which "updates itself" whenever the value is observed. PullGauges are also used for the built-int GHC metrics.
• The JSON parts are still in there.

It's most likely that it's not yet fully ready for a pull request (especially since the server part is not separated out yet), but I'd appreciate feedback to polish it up, if you think this is going in the right direction.

Disclaimer: This is pretty much my first "real" Haskell code (apart from playing around and toy examples), so there are probably quite a few rough edges ;)

[tibbe]

That sounds cool. I'm a bit snowed under with some hashable/unordered-containers excitement at the moment so it might take a while before I can look at this.

[gseitz]

Sure, I'll keep on add things. If nothing else, I get some practice with Haskell ;)

[tibbe]

Ooops. Closed by misstake.