Migrate to http://micrometer.io/

[jdubois]

Pivotal did their own "dropwizard-metrics" library, and is pushing it with Spring Boot 2. This is called http://micrometer.io/ and we are basically forced to use it.

Yes I sound a bit harsh, because I'm very happy with Dropwizard Metrics: we've been using it for years with great success, and did some custom development on top of it. So we'll need to replace it with a far less popular project, with less contributors, and we'll need to test and re-code everything :-(

The testing part is particularly annoying, as we have some major websites using JHipster with Dropwizard Metrics, running without any issue.

Another problem is that it doesn't support Graphite yet, and we've had this integration from the start.

So we need to migrate, test everything, and maybe lose some features. Lots of work ahead!

[deepu105]

Did they remove support for Dropwizard?

Thanks & Regards, Deepu

On Sat, Feb 10, 2018 at 12:18 PM, Julien Dubois notifications@github.com wrote:

Pivotal did their own "dropwizard-metrics" library, and is pushing it with Spring Boot 2. This is called http://micrometer.io/ and we are basically forced to use it.

Yes I sound a bit harsh, because I'm very happy with Dropwizard Metrics: we've been using it for years with great success, and did some custom development on top of it. So we'll need to replace it with a far less popular project, with less contributors, and we'll need to test and re-code everything :-(

The testing part is particularly annoying, as we have some major websites using JHipster with Dropwizard Metrics, running without any issue.

Another problem is that it doesn't support Graphite yet, and we've had this integration from the start.

So we need to migrate, test everything, and maybe lose some features. Lots of work ahead!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jhipster/generator-jhipster/issues/7100, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDlF7GVcSJeDgPZMTN-COQnEp9x5791ks5tTXsBgaJpZM4SA5QM .

[jdubois]

It still works, but it's not good to have 2 competing libraries:

• more stuff in the classpath
• more configuration

Besides, micrometer copied the Dropwizard configuration classes, so for example they have the same "@Timed" annotation. You would then need to have a look at the import statements to know which one you are using.

[deepu105]

@jdubois seems like our configuration screen is broken as well with spring boot 2 due to api url & response schema changes I guess

[jdubois]

Yes they changed the JSON format. It's more annoying for the Registry as we will need to support both.

[PierreBesson]

@jdubois Micrometer seems to have a graphite exporter in the latest version (http://micrometer.io/docs/registry/graphite). It even has exporters for influx, statsd and cloud services like datadog and cloudwatch. In this regard, it is fully replacing a lot of what we did with configuring exporters and it even supports more systems.

One thing that will need some work though is to create a "log exporter" for the console.

[jdubois]

Yes I agree it's going to replace a lot of what we already have. I'm mostly worried by the testing part (Dropwizard Metrics is battle-tested on lots of high volume apps) and size of the community (Micrometer is mostly done by one person, and from my understanding he is paid by Pivotal to do this - so it doesn't guarantee long-term support of the project). Anyway, yes we need to migrate, there's no doubt about it, so I'm not going to do a rant on this. I also agree that the "log exporter" is missing, and also we need to do again our metrics screen (in AngularJS, Angular, React, and on the Registry, maybe also the JHipster Console? that sucks...). But that's all doable.

[pascalgrimaud]

I wonder how we can keep compatibility with the jhipster-registry:

• I suppose we need to migrate the jhipster-registry to Spring Boot 2 as well
• then, this new registry will only work with Spring Boot 2 applications, and not 1.5.x ?

[PierreBesson]

@pascalgrimaud, as the registry with sb2 will embed the same versions of eureka and the config server it should be perfectly compatible with both sb1 et sb2 apps.

[pascalgrimaud]

@PierreBesson : good for the backend, but I think about the front with the metrics

[jdubois]

I've played with micrometers quite a lot, and I'm not convinced for the moment, let me explain in more details.

Indeed, it replaces many things we do. At first sight, this would be an easier and nearly no-configuration integration. But if you start really using it, a lot of limitations occur:

• No console exporter, as @PierreBesson found out
• Integration of "advanced" stuff like caching is quite cumbersome. There's not much documentation on this, and it seems that you need to configure this manually - in my tests, that's more code, giving less information, and I can't get any metrics at the end :-(
• Same thing for the JVM or the datasource, we have less metrics, and some stuff just don't work out-of-the-box
• The worst part is for the Web metrics. It gives some out-of-the-box metrics on your REST endpoints (it's lazy-loaded, so you need to use them in order to see them), but they use "tags" for this. We currently have metrics per endpoint, and even per Spring bean method, which I find better-suited to my needs. So I needed to use an @Timed annotation again - that's the same name as with Dropwizard Metrics, but you have much more configuration to give here so it's more annoying (I'm considering doing our own annotation to reduce the boilerplate)
• The metrics are really strange. There is never a "min" value (why not??), and I haven't been able to use the percentiles. The percentiles aren't documented, so I'm guessing they don't really work, and the API I used is cumbersome. The worst part, of course, is that I couldn't make it work (all values are empty), and for me percentiles are a lot more important than the average or the max value.
• The REST endpoint is useless for us. You need to do one request per metric you want, and even per tag (for the MVC part) - so for our metrics screen, that means you would do a lot of requests (when today we only do one). As this screen gets refreshed quite often, this is just not possible for us. To use this efficiently, we would need to do our own REST endpoint - that's doable (in fact I already did it), but it's quite a lot of code, and that's not really good for our users (we would need to put it in our server lib, and then that would specific to JHipster)

Last but not least, I still have issues about Micrometer not being battle-tested yet, and being done by only one person paid by Pivotal (so it can disappear very easily).

So at the moment I'd rather stay with Dropwizard Metrics for some time, at least one year. This will give them time to:

• Build a real community around the project
• Have some real-life tests
• Have a better documentation
• Clean up and improve the API

For me it's a matter of time until we migrate, but it's just not the time.

So unless anybody disagrees, I'll close this.

[deepu105]

I second @jdubois lets stick to Dropwizard for now as we have already too much on our plates and this definitely isn't mature enough to replace what we have. We can try to exclude the micrometer related libs from our dependencies to avoid increasing war size.

[jdubois]

Oh yes we need to disable it: it's enabled by default, so there's a performance hit. I'm quite surprised they put on it by default for everyone.

[jdubois]

I've excluded the library. Let's close this for the moment - I'll reopen if anybody wants to discuss this further, of course

[PierreBesson]

Sorry for being late to reply. Although I was excited about it at first, considering the arguments put forward by @jdubois, I think it's wise to wait a bit on this one. It's definitely not simple to recreate the advanced setup we had with dropwizard. Let's make sure that monitoring still works on SB2 and then maybe we can migrate to micrometer bit by bit.

[jkschneider]

I'd love to help out in getting you over the migration hump. I do agree it's best for you to push something Boot 2 compatible and then we can sit down and address this.

Hope I can clear up a few understandable misconceptions here, though. First, a little "why" is in order I think. I promise I don't want to argue for Micrometer's relevance, but I think a little background helps shed light on some of the issues you've identified:

Why another metrics library?

Micrometer is a dimensional first metrics collector that can map dimensional metrics to hierarchical names to continue to serve older monitoring solutions like Graphite. It is important for us though to think about how best to serve the wave of dimensional systems (think Prometheus, Datadog, Wavefront, SignalFx, Influx, etc). This is especially true considering the fact that Boot major releases don't come along every day ;)

Dropwizard has a plan to "go dimensional" with tagging support in 5.0. This is of course going to require a change to pretty much all dropwizard-based instrumentation anyway. But simply adding tags isn't close to enough. As you export to more and more monitoring systems, the importance of the structure of names and data becomes more important. So Micrometer builds in concepts of naming convention normalization, base unit of time scaling, and support for proprietary expressions of structures like histogram data that are essential to make metrics actually usable in the target system.

There's a lot more, but I'll stop there as much of the rest isn't relevant to the issues brought up here.

Issues

No console exporter, as @PierreBesson found out

It is possible to configure a console exporter, if you must. As more and more instrumentation is added, seems like this is less and less desirable though. I hope it's understandable why first-class support for this wasn't a priority.

Integration of "advanced" stuff like caching is quite cumbersome. There's not much documentation on this, and it seems that you need to configure this manually - in my tests, that's more code, giving less information, and I can't get any metrics at the end :-(

I'd like to hear more. Boot 2 autoconfigures cache metrics via CacheMetricsAutoConfiguration. Documentation is here. This has been strictly additive, with more detailed metrics than existed previously in Boot based on the cache implementation (note that Guava support was removed in Boot 2 in favor of Caffeine).

Same thing for the JVM or the datasource, we have less metrics, and some stuff just don't work out-of-the-box

I'd like to hear more. JVM and datasource metrics are autoconfigured in Boot 2. Again, there is more instrumented in Boot 2 than there was in Boot 1.5, and we've carefully combed Prometheus, Spectator, and Dropwizard for useful JVM metrics. What have we missed?

The worst part is for the Web metrics...

When you export a web metric to a hierarchical system like graphite, you do get an individual metric for each endpoint, because Micrometer flattens dimensional IDs into a hierarchical name based on the configured HierarchicalNameMapper. By default it would look something like:

httpServerRequests.method.GET.response.200.uri./people/{id}

Manually assigning a hierarchical name to each endpoint seems unnecessary for Graphite, and is actively harmful when shipping these metrics to dimensional systems that rely on the tag data.

The metrics are really strange. There is never a "min" value (why not??)

Experience with Netflix SREs was that min was rarely useful for the primary purpose of metrics data: alerting and scaling. We do have an open issue on min specifically that has been pushed to 1.1.0 where there is some rumination over whether to make it a default stat shipped at the cost of extra footprint or an opt-in thing (leaning towards the latter). Notably, opencensus didn't consider it worthy originally either. We're not alone here.

The percentiles aren't documented, so I'm guessing they don't really work, and the API I used is cumbersome.

Documented here and here. Note that client-side percentiles are non-aggregable and so generally not all that useful as you add more and more dimensions. Aggregable percentile histograms are covered in the same doc sections. Bottom line though: I believe it best to leave it up to the user to add percentiles or histograms via one of these properties for the metrics they need those for.

The REST endpoint is useless for us... To use this efficiently, we would need to do our own REST endpoint - that's doable...

I think this is an accurate assessment for your use. To be honest, you essentially have a micro monitoring system in your UI, and that is deserving of its own exposition.

It was trivial to provide a single REST endpoint that listed all metrics in Boot 1.5, but then we only had counters and gauges, and both were hierarchical... As we matured into more complex types like timers and went dimensional, it became clear that there was no way to output all this information in a way that satisfied UIs like yours. Even for a dimensional counter, do we display an aggregate for every permutation of tags? Flattening to hierarchical names for brevity, this turns into something like this:

http.server.requests.method.GET.response.200.uri./foo=100
http.server.requests.method.GET.response.500.uri./foo=1
http.server.requests.method.GET.response.200.uri./bar=5
http.server.requests.method.GET.response.400.uri./foo=1

# and now the aggregates...
http.server.requests.method.GET=107
http.server.requests.method.GET.response.200=105
http.server.requests.method.GET.uri./foo=101
http.server.requests.response.200.uri./foo=100
http.server.requests.response.500.uri./foo=1
http.server.requests.response.200.uri./bar=5
...

As you can see, this quickly becomes untenable. But if you know that your UI is only interested in http throughput by URI irrespective of the method, status, etc then the output can be substantially curtailed.

being done by only one person paid by Pivotal (so it can disappear very easily).

So that's me. Yay. I guess this a bit of an affront to the dozens people that contributed code and the many more that provided input along the way. It wouldn't be what it is without them. I understand the spirit of the concern though: it's only been around in development for a year and it doesn't have a million stars on GH.

Boot 2's metrics are wholly reliant on Micrometer, so it would require a major Boot release to remove it. That simply isn't going to happen "very easily".

And then there is this guarantee, which Micrometer is also subject to.

[deepu105]

@jkschneider thanks for clearing up the concerns. We might have sounded a bit annoyed since we already had too much stuff on our plate and this wasn't something we wanted to do (as there was no real value for us to do the migration at the moment) but in the long run we would like to use what is provided as default by SB so that we can remove our custom code and a competing library. But as @jdubois highlighted our Dropwizard based implementation is battle tested and provides in-depth metrics. We have some custom code to generate stats for Jcache based cache etc. So unless we can get at least a closer implementation our metrics screen would look much less attractive for end users.

But yes we will migrate to this at some point but for now, the effort of migration is too much for us.

[jkschneider]

We have some custom code to generate stats for Jcache based cache etc.

Spring autoconfigures JCache metrics now, so we should luckily be able to remove that custom code eventually.

So unless we can get at least a closer implementation our metrics screen would look much less attractive for end users.

Clearly we are going to have to provide an exposition to satisfy your metrics screen. There is absolutely no reason you should lose information here though.

our Dropwizard based implementation is battle tested

As long as you don't attempt to disable Micrometer or remove it from the classpath, I don't think there is any problem allowing them to co-exist. Force disabling it would mean JHipster is removing a key Boot 2 feature and makes it less attractive, I believe. But it looks like we've settled on addressing the root cause of not monitoring Hikari so that such exclusion is not necessary?

[jdubois]

Just to make it clear @jkschneider : there's nothing against you when I say this is done by one person paid by Pivotal. It's great that you work on Micrometer, and that Pivotal pays you to do this. However, I know that it can make an Open Source project less stable. That's what happened recently to Groovy and Grails, and I can name so many more Pivotal products (SpringSource dm Server, Hyperic, Spring Webflow....), all which I loved and were simply stopped by Pivotal. As soon as you don't bring money, the company can kill the project - and that's normal. Many other companies do the same, like Google for instance - and killing AngularJS really hurt a lot of people. So yes I would rather have had a "vendor-neutral" monitoring system (just API support, like Spring has always done before), or a Pivotal contribution to Dropwizard Metrics as it's the de-facto standard. Now that's not such a big deal: I think we can all agree here that the biggest issue is the "aggregate REST endpoint", and that's something we should be able to port.

Please note that's extremely important for us: the vast majority of our users don't want a third-party monitoring tool, they want this inside their application. I guess we have 2 populations using it: people who want to have something cheap (including software vendors who deploy their apps on customer sites), and people who don't want to rely on their ops team (I originally coded this while at a big French bank, where we couldn't have access to the monitoring tools, as licenses were too expensive).

[cbornet]

@jkschneider you know the subject far better than us. Maybe you could help us with the migration with a PR ?

[jkschneider]

So yes I would rather have had a "vendor-neutral" monitoring system (just API support, like Spring has always done before), or a Pivotal contribution to Dropwizard Metrics as it's the de-facto standard.

All concerns about Pivotal sponsorship of Micrometer understood, I think unfortunately Dropwizard Metrics is fatally flawed for the modern monitoring landscape. It isn't just a tweak or two away from being right. I wish this wasn't so as much as anybody. I really just want to move up the stack and work on automated canary analysis, predictive autoscaling, dynamic alerting, etc. But we simply had to address the deficiencies of basic instrumentation first :/

...don't want a third-party monitoring tool, they want this inside their application

I'm totally sympathetic to your UI, and think it is a useful component. And we are in the same boat: spring-cloud-dataflow has a custom UI like this, we need to replace Hystrix Dash with a similar custom UI component sourced from metrics data, and @smaldini wants a custom UI for reactor metrics. Just want us to be honest about the fact that each of these presentations is just a micro monitoring system tailored to specific metrics.

@cbornet I'll help with the implementation for sure!

I want you to be prepared for a shift in your population as well. With increasingly popularity of open source monitoring tools like Prometheus, the expensive license argument is vanishing quick. Some population is surely going to want to use JHipster's UI for instance-level inspection and also ship to a full-blown monitoring system.

[jdubois]

Oh yes 2 very good points:

• Dropwizard Metrics is dying. I don't know why technically, but from a community perspective it's clear that they are not very active anymore.
• And yes we will need your help @jkschneider because this is going to take us forever to migrate otherwise.

[georgeplaton7]

Let us know @jkschneider and @jdubois when the migration is planned :-) so we can look ahead for the migration. Thanks in advance!