Unleash Edge metrics - would like to have appName flow through from client app -> Edge -> metrics exposed by Prometheus endpoint

[YayBurritos]

Describe the feature request

This request started from a discussion in the Unleash public Slack help channel on March 5, 2024...

In looking at the metrics provided by Unleash Server at /internal-backstage/prometheus, it appears that the feature_toggle_usage_total metric doesn't have any items reported by Unleash Edge (based on the appName label values). I would have expected Edge to communicate this info similar to how the Unleash Proxy does it. I see plenty of feature_toggle_usage_total metrics with appName="{our Unleash Proxy}" , but nothing from Edge.

The http_request_duration_milliseconds metric has items where appName="unleash-edge" , which is good. But would be nice to know which toggles are being used by Edge as well. Am I missing something in the Edge configuration that can make this happen? Thanks!

@ivarconr responded:

Edge is not the consumer of the toggle. You would look for the actual appName in the metrics.

I then questioned:

1. How do I pass the individual "consumer" names to Edge so that Edge will provide that for the metrics? The "consumer" would be the application that is calling Unleash Edge.
2. Why does the Proxy send the configured UNLEASH_APP_NAME to upstream Unleash to be included in the metrics since it's not the true "consumer"? Proxy and Edge seem to differ in this regard.

@ivarconr responded to those two questions:

1. Edge and unleash have seperate Prometheus endpoints per instance. It does not make sense for edge to pass Prometheus data upstream.
2. Because proxy was a bit strange. It acted more like a regular SDK

Ivar asked for clarification as to what we're looking to accomplish (what problem we're trying to solve). That info is in the Background section below.

Background

We have multiple applications (and multiple environments for each app) using Unleash (Edge and Server) for feature toggles. We build various dashboards in Grafana using Prometheus (and other) data.

For a given feature toggle, it can be helpful to know which applications are actually using it...and possibly how often the toggle is being checked...

Potential Use Cases

• Maybe we have a code issue where toggle A is being checked an unreasonable # of times relative to others
• Or toggle B isn't being checked at all (maybe we should remove this toggle...or we instead realize we need to update one of our apps to start using it...or it's no longer being used due to a code regression and we need to fix this)
• Or we'd like to remove toggle C altogether, but we realize it's being used by apps 1, 2 & 3, meaning we'll have to update the code for all 3 apps and deploy new versions of each before we can really remove the toggle from Unleash

Those are just some ideas. We can get the metrics for both Edge and Server into Prometheus and having a single place to view them (Grafana, where most of our other metrics surface) is preferable over having to poke around in the Unleash UI (IMO).

Solution suggestions

I've been looking at the toggle metrics in Unleash UI. Seeing Applications that can be used to filter. Those names align with our applications and do seem to flow through from Edge to the metrics displayed in the Unleash UI. Having those same names as the appName label for the feature_toggle_usage_total metric would likely allow us to address the use case(s) outlined above.

I appreciate the response. If there's anything I can do to help, please let me know. Thanks!

[daveleek]

Hello @YayBurritos and thank you for your contribution! We have recorded this as an idea input for future planning. We might opt for a different solution to the issue, one that matches our architecture in a better way.

[YayBurritos]

Just to add a bit more context here for our specific situation, we have two client applications using Edge:

• React web app using unleash-proxy-client
• Mobile app using proxy-client-react

Each application is instantiating an UnleashClient and passing in a unique appName (so let's just say web-app-123 and mobile-app-123 for the sake of argument).

I understand that it may not make sense for Edge to pass this along to Unleash Server. So to satisfy this particular issue/feature request, maybe the following alternative would work:

• Modify Unleash Edge to expose a feature_toggle_usage_total metric (just like Unleash Server does) with an appName label that matches the appName used to instantiate the UnleashClient.
• So when looking at the Edge metrics at /internal-backstage/metrics, we could see something like this:
  • feature_toggle_usage_total{toggle="SomeToggleName",active="true",appName="web-app-123"} 5
  • feature_toggle_usage_total{toggle="SomeToggleName",active="true",appName="mobile-app-123"} 10

That would allow us to know which Edge clients are using our toggles. Thanks!

[chriswk]

So, are you saying that you don't see feature_toggle_usage_total{toggle="SomeToggleName", active="true", appName="web-app-123"} in unleash-server, because as I remember it, Edge forwards appNames it has received metrics for up to unleash-server.

So any usage of Edge by web-app-123, should be reflected in the feature_toggle_usage_total in the unleash-server that Edge is connected to.

I can see the value in being able to see feature_toggle_usage_total on the edge /metrics endpoint as well though, so you could do some grafana magic showing what percentage of traffic is going via edge.

[YayBurritos]

The only appNames that we see in the feature_toggle_usage_total metric from Unleash Server at the /internal-backstage/prometheus endpoint are those that are reported from our backend (dotnet) SDKs.

We aren't seeing any appNames that our frontend (JS) SDKs pass to Edge. And Edge metrics currently don't include a feature_toggle_usage_total metric. (NOTE: We're currently using Edge 17.1.0).

I can see the value in being able to see feature_toggle_usage_total on the edge /metrics endpoint as well though, so you could do some grafana magic showing what percentage of traffic is going via edge.

Yes - that would be great. We could certainly query the feature_toggle_usage_total metric from both Unleash Edge (at /internal-backstage/metrics) & Server (at /internal-backstage/prometheus) and display the information on a Grafana dashboard to better understand our toggle usage if that metric was made available by Edge.

[chriswk]

Hi @YayBurritos - a bit of a delayed answer, but with https://github.com/Unleash/unleash/pull/7210 - Unleash should now include metrics coming from Edge, so you should see feature_toggle_usage_total update based on usage from Edge as well. In addition, I've made #469 to track that we add feature_toggle_usage_total to edge's /internal-backstage/metrics as well.

[YayBurritos]

@chriswk : Is there an anticipated Unleash version for these changes? Looks like Unleash v5.12.7 was created a few hours ago, but this change doesn't appear to be included. Just wondering on the process/timing of PRs being merged to main and then becoming part of an official release. Thanks!

[ivarconr]

@YayBurritos it is included in v6.

(older versions will usually just get patches / bug fixes)

[YayBurritos]

Thanks @ivarconr!

[YayBurritos]

@ivarconr , @chriswk : Sorry guys - another question. How soon after new Unleash versions/releases are the Helm charts updated?

[YayBurritos]

Hi @YayBurritos - a bit of a delayed answer, but with Unleash/unleash#7210 - Unleash should now include metrics coming from Edge, so you should see feature_toggle_usage_total update based on usage from Edge as well. In addition, I've made #469 to track that we add feature_toggle_usage_total to edge's /internal-backstage/metrics as well.

Hey @chriswk - Just wanted to give you a quick update...

We recently upgraded to Unleash Server v6 and now see our appNames flowing from Edge to Server and appearing in the feature_toggle_usage_total metric from Server. This is very helpful!

That said, we do still see value in also being able to see the feature_toggle_usage_total metric exposed by the Edge /internal-backstage/metrics endpoint. It is less of a priority for us now that we have the data from Server, but still would be a nice-to-have.

Thanks so much!