Support Apache Camel

[jkschneider]

See https://github.com/laurikimmel/camel-metrics See https://brunonetid.github.io/2017/11/27/camel-prometheus-openshift.html

cc / @bjartek

[bjartek]

I have to look internally at our Camel stuff but could this be as «simple» as Creating some producers? http://camel.apache.org/metrics-component.html

How do we handle labling? We have to parse the to string somehow?

[jacovt]

As I see it, there might be 2 parts to this (unless I am completely missing the boat):

1. Implement it into the metrics component as a producer.
2. Create a MeterBinder implementation that simply exposes all the camel metrics that are already exists in JMX. Something similar to what is being done for Tomcat: https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/tomcat/TomcatMetrics.java

I think we might have to do a little fancy naming footwork with the JMX metrics to make it easier to graph in something like Grafana.

Thoughts?

Thanks, ---Jaco

[ohr]

Thoughts:

• next to the Metrics producers there are also MetricsRoutePolicy(Factory) and MetricsMessageHistory(Factory)
• additionally, Camel has indeed some MBeans that expose performance metrics (sub-classes of org.apache.camel.management.mbean.ManagedPerformanceCounter, mainly ManagedRoute, ManagedProcessor, ManagedCamelContext). These metrics are collected without any 3rd party dependency and were primarily meant to be exposed via JMX only. Having MicrometerRoutePolicies and MicrometerMessageHistory in place should make extracting Gauges from Camel MBean attributes actually redundant IMO.
• Support for Camel - where? It appears to me that some of the binders that already exist and requests like #533, #526, #512, #343 ... go beyond of what I would expect from a micrometer-core lib.

[jkschneider]

@ohr Biggest question you raise is also a question of mine: where should this live?

The best possible scenario (for us) is that instrumentation lives in Camel itself (similar to what was done for rabbitmq-java and hikaricp), but I know that isn't always possible. The binders we've included in the core module so far are chosen deliberately for their long-established API stability.

We could of course create a new module for each new integration, micrometer-httpclient, micrometer-okhttp, etc. And of course Prometheus has decided to do this with simpleclient_xxx for lots of different integrations. Developer experience is kind of rough though: first there is the discoverability problem of just knowing what bindings exist, and then there is the problem of including additional dependencies and keeping their versions aligned. I can see a future where we have io.micrometer.contrib:micrometer-camel or something like that if we want binders for products that have less stable APIs (Camel may or may not, I don't know either way). We just haven't encountered that yet. The line may always be a bit blurry.

[jkschneider]

MetricsRoutePolicy is already directly tied to Dropwizard Metrics. We could certainly build timers off of the onExchangeBegin/onExchangeDone events, but Camel already appears to have a bit of a leaky opinion here.

[bjartek]

Could we host it in the micrometer org but name it micrometer-camel?

[checketts]

@bjartek Jon did mention that option when he referred to the Prometheus simpleclient_xxx layout. What I've seen in that project is that after a while they stopped accepting additional modules.

The only problem I see with leaving them in core is the need to be careful that the camel dep remains optional and class usage is kept within that binder.

Is anyone here a camel contributor that could try putting the micrometer support directly in Camel?

[ohr]

I contributed a couple of things to Camel already so I could prepare a pull request in Camel for micrometer support (following the pattern from camel-metrics. @davsclaus: any objections?)

Note, however, that Camel has already a LOT of components that integrate 3rd party libs, and a considerable percentage of them have a tendency to become outdated over time (i.e. not catching up with new versions and features of the 3rd party libs being covered). So, depending on further development here (particularly when introducing breaking changes!), this would need ongoing attention.

[jkschneider]

@ohr Being a Pivotal sponsored project, Micrometer is bound to 3 year support for major releases (no breaking public API changes inside a major release) and 1 year for minor. Realistically, it's probably longer for Micrometer since Spring Boot 2 depends on it, and its life expectancy is probably greater than 3 years.

[rpalcolea]

Hi! we are doing something similar with MBeanServer to extract kafka consumer/producer/streams metrics:

https://docs.confluent.io/current/kafka/monitoring.html#global-request-metrics https://docs.confluent.io/current/streams/monitoring.html#accessing-metrics-via-jmx-and-reporters

While it would be nice to have this integrated in micrometer-core, I understand the support of breaking changes (like metrics names, etc.)

What about extracting the MBeanServer logic from the Tomcat implementation in a way that users could extend an abstract class (perhaps?) and just provide details of what they want to report as metrics?

[davsclaus]

@ohr yeah that would be lovely. There is a JIRA ticket already: https://issues.apache.org/jira/browse/CAMEL-11600

I disagree a bit on your about components gets outdated, we upgrade to latest versions as much as possible, and include all PRs that people contribute, and we overhaul and redo components on major new versions like elasticsearch. The camel-spark component however indeed needs an upgrade to v2 or to do a camel-spark2 component. Use the JIRA voting to help prioritize and also contribute as you have done.

[jonmcewen]

@ohr any progress on this? I'm looking to upgrade to spring boot 2, and currently have dropwizard metrics from all camel routes, which I don't want to lose!

[jkschneider]

@jonmcewen We want a generalized solution for Apache Camel metrics still, but there is no reason you have to lose Dropwizard metrics upgrading to Boot 2. Suppose you are shipping your dropwizard metrics to graphite (similar story for other dropwizardy-backends). You can either:

1. Wire your own @Bean GraphiteMeterRegistry using GraphiteMeterRegistry(GraphiteConfig config, Clock clock, HierarchicalNameMapper nameMapper, MetricRegistry metricRegistry). Supply your existing DW MetricRegistry.

2. Allow Spring Boot to create the DW meter registry, then @Autowire GraphiteMeterRegistry and call getDropwizardRegistry() on it to retrieve the DW MetricRegistry that Boot has created.

[jonmcewen]

@jkschneider great. Is there any further documentation or example code for this?

[ohr]

Slowly progressing, been busy recently.... Work will be tracked in https://issues.apache.org/jira/browse/CAMEL-11600.

Some dropwizard concepts used in Camel cannot be mapped 1:1, such as setting Gauge values via a producer. On a side note, I'd also prefer to have a native JMXRegistry (not via DropwizardRegistry and JmxReporter) just to avoid having a second metrics library around. Opinions? I also had to write a proper JSON renderer as I couldn't find one in micrometer.. @jkschneider: would it make sense to move it here?

[jkschneider]

such as setting Gauge values via a producer.

@ohr if you could show me a Dropwizard line in camel that uses this, I'll show you how to do it with a Micrometer Gauge.

I'd also prefer to have a native JMXRegistry

I could see us doing this at the same time as we define a generalized mechanism to pre-aggregate certain dimensions. This work was originally required for CloudWatch, but we've since found that it is highly applicable to JMX as well.

I also had to write a proper JSON renderer as I couldn't find one in micrometer

So far we've just been using String concat, but I wouldn't object to an internal utility that makes this easier, provided that it doesn't involve a Jackson-like library. Not that I dislike Jackson, but don't want Micrometer to poison the classpath with opinions on libraries like that.

[ohr]

A Camel Dropwizard gauge maintains its "private" value that is actively set when an exchange passes the producer:

   Object subject = exchange.getIn().getHeader(HEADER_GAUGE_SUBJECT, Object.class);
   if (subject != null) {
      camelMetricsGauge.setValue(subject);
   }

Micrometer observes a weak reference, so I guess that simply having the Camel route to modify the target of that reference is preferrable.

With your remarks, I skip JMX stuff for the time being. In fact I wrote a Jackson-based JSON serializer (following the pattern of the camel-metrics module), so I keep it over there.

[ohr]

CAMEL-11600 was resolved. So this could be closed as well.

[izeye]

As @ohr stated above, this looks resolved via the Micrometer component which lives in the Apache Camel repo.

[shakuzen]

As mentioned by @izeye and @ohr, Micrometer support is available in Apache Camel since Apache Camel 2.22.0