MicroProfile Metrics Not Providing Solution For Multiple Apps on Same Server

[dbnex14]

There is no clear instructions how to add application name or context root to Microprofile metrics. In Spring, you get this by default. If this is possible in Microprofile, then the documentation should provide examples to show how to do this rather than writing an essay about it. I am refering to the section 3.4 in this document https://download.eclipse.org/microprofile/microprofile-metrics-2.3/microprofile-metrics-spec-2.3.html#app-servers. I am also not able to find any example about this online.

All examples show default metrics url

http://myServer/metrics

instead of

http://myServer/app1/metrics
http://myServer/app2/metrics
http://myServer/app3/metrics

Can we have clear example how to set this up and get desired urls. As it is now, the Microprofile metrics is of no use to us as we have several apps running on same server and it is impossible to see/distinguish btw them.

[Channyboy]

Hello,

Differentiating multiple applications deployed on the same server is accomplished with the _app tag. Spec does not define context-root support.

Once received by Prometheus (and by extension Grafana), it's the use of tags which distinguishes the metric.

The value of the _app tag should be passed by the application server to the application via a MicroProfile Config property named mp.metrics.appName. It should be possible to override this value by bundling the file META-INF/microprofile-config.properties within the application archive and setting a custom value for the property mp.metrics.appName inside it.

In your case, you will need to bundle each application with a META-INF/microprofile-config.properties containing the mp.metrics.appName config. e.g. mp.metrics.appName=appName

[dbnex14]

Thanks but I'm talking about web app here. I don't have META-INF, I only have WEB-INF, and I added to it new file microprofile-config.properties containing the mp.metrics.appName=app1 as you suggested.

After that, I tried using few requests and accessing url with app name in it like

http://mysetver/app1/metrics

However, I get error that url is not valid.

The metros is still showing at

http://mysetver/metrics

So, it isn't working.

—

On Tue., Nov. 2, 2021, 12:02 p.m. David Chan, @.***> wrote:

Hello,

Differentiating multiple applications deployed on the same server is accomplished with the _app tag. Spec does not define context-root support.

The value of the _app tag should be passed by the application server to the application via a MicroProfile Config property named mp.metrics.appName. It should be possible to override this value by bundling the file META-INF/microprofile-config.properties within the application archive and setting a custom value for the property mp.metrics.appName inside it.

In your case, you will need to bundle each application with a META-INF/microprofile-config.properties containing the mp.metrics.appName config. e.g. mp.metrics.appName=app1

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/microprofile-metrics/issues/652#issuecomment-958075419, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCZTX5BEBERE7LANHMTQNLUKA7UJANCNFSM5HGZNTVQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Channyboy]

There is no context-root support.

It will always be http://host:port/metrics

Do you see your metrics on /metrics with the _app tag?

[dbnex14]

I do not see_app tag anywhere and I do not know how to set _apptag, let me explain from beginning to end what I did, maybe that will discover the problem.

In my .war project (which is packaged in an .ear), I have src/main/java/webapp/WEB-INF folder. Within that folder, there was NO microprofile-config.properties file, so I added one I added following content to that file:

mp.metrics.appName=app1

I have also annotated one of my methods called by my REST controller with metrics annotation like:

@Counted (name="getCar-count", displayName="number of getCar calls", description="blahblahblah")
public Car getCar() {...}

Then I rebuild my project, deploy to WAS Liberty or Open Liberty server (these have <feature>microProfile-3.2</feature> feature) and issue a couple of requests to the controller that calls the above getCar() method.

I try accessing following 2 urls:

1. http://myServer/app1/metrics, I get below error and as you replied this is not supported by Microprofile (although the section 3.4 of your documentation at https://download.eclipse.org/microprofile/microprofile-metrics-2.3/microprofile-metrics-spec-2.3.html#app-servers is not clear about this as it is not clear how to setup _app tag)

   Error Message: javax.servlet.ServletException: java.io.FileNotFoundException: SRVE0190E: File not found: /metrics
   Error Code: 404

2. http://myServer/metrics, I see below:

# TYPE application_my_domain_app1_getCar_count_total counter
# HELP application_my_domain_app1_getCar_retrieveCard_count_total Total number of getCar calls serviced
application_my_domain_app1_getCar_count_total 2

As you can see, I do not see any tag _app=/app1 as your above document states I should see. That would help; however, generic url for entire server without having app name withing the url is not practical at all as most servers host multiple applications so it is very difficult to figure out what is coming from which app. I think Spring realized this and added nice separating by the app in the url by default like http://myServer/actuator/app1/metrics which then allows us to focus on actual app metrics instead of focusing on filtering metrics we need from entire server.

[Channyboy]

Hello, You should still be able to create a META-INF folder within your WAR.

Since the format you are using would be Prometheus/Open Metrics format it would be expected that the user would be pulling dating with a Prometheus server. Furthermore, Grafana would typically be used as the visualization platform for displaying the time-series metrics stored in Prometheus. And it is typically expected that the Prometheus/OpenMetric metrics are available at a simple /metrics endpoint.

Once metrics are retrieved from all target endpoints it will be stored in the Prometheus database together. Not knowing if a multiple application scenario on a single server are of the "same" applications or of varied applications, name clashes can occur. Multiple context-roots would not solve this problem. To appropriately differentiate metrics, the use of tags are necessary. That was when the mp.metrics.appName MP config value was introduced.

Within that folder, there was NO microprofile-config.properties file

Right, it is not expected to be there normally, this is one of the use cases of the MP Config technology. Most MP technologies rely on usage of MP Config. See the MicroProfile Config Technology for more information.

Additionally, the Microprofile Metric specification pertains to information about how to use the Metric technology itself and does not typically explain the usage of other MicroProfile technologies. It does seem like we do specify the file and location that we expect the MP config value to be discovered. However, if there is an issue with the wording, feel free to open an issue or PR with suggestions on how to address the vagueness of the paragraph. It's always good to have fresh eyes!

[dbnex14]

OK, I created new META-INF in my war project's Deployed Resources > webapp folder.

Then I created file microprofile-config.properties within META-INF folder

 My .war project root
     | - Deployed Resources
          |- webapp
               | - META-INF
                    |- microprofile-config.properteis 

, and added following content to that properties file:

mp.metrics.appName=app1

Then I rebuild my app and redeploy and now I see _app="app1" in my metrics data, so /metrics endpoint is good.

However, I still dont see this _app tag show in my /health endpoint. All I see is

{"checks":[],"status":"UP"}

So, again, same issues, if I have 10 apps like app1, app2, app3, .... app10 on same server, how do I know which one is up and which one is down?

[Channyboy]

The /health endpoint is related to the MicroProfile Health technology, which is separate from MicroProfile Metrics.

I'm not involved on MicroProfile Health, but here is my take. There is an option to build the health checks with optional (meta)data which work somewhat like tags like in Metrics. However, there is no application level wide MP Config value that can be set for this, this will need to be done programmatically in the application. This "pain-point" is expected as MP Health typically is not a natural fit for a multi application deployment in an off-cloud environment.

The purpose of MicroProfile Health is to indicate if a microservice is alive or dead(which would typically be a single application running in a container on your cloud platform) to instigate the container management system to restart the container or not. Or if it is ready or not to receive traffic from other microservices.

The accommodation by MicroProfile Metrics for multi application server deployment is due to the fact that this technology can fare better in an off cloud/container environment. Who doesn't want metrics? And it's easily plugged into Prometheus and Grafana which doesn't have to run on the cloud. Health on the other hand is a little be more niche.

If you want to find out more, you can ask the folks over at MicroProfile Health

[dbnex14]

Thank you

[dbnex14]

I still think this is design flaw. The _app tag is more of a hack and not helpful that much. All it does is that it moves focus from "checking your metrics" to "sifting through metrics to identify right ones from your application".
It is smell really and requiring to use Prometheus and/or Grafana to be able to do what you need to do is a sign of not good solution. I think Micropfofile should provide this feature out of box as Spring does. I think you guys made assumption that everything will be microservices one day but microservices are not a solution for many real time scenarios but are rather overhead. We have seen many articles on the topic of rise and fall of microservices. But not to introduce new topic, microservices or not, this should be a feature provided out of the box.