support for telemetry without mixer

[ihiteshwar]

As mixer is being deprecated, what is the plan to support telemetry import into the newrelic.

[dinup24]

We are planning to upgrade our cluster to Istio 1.5.1. Currently, we are using this mixer based newrelic adapter. What would be the recommended approach to send metrics to newrelic with 1.5.1. Looks like there is a provision to install Mixer with 1.5.1 - should we take that approach? Or wait for the new adapter/agent for telemetry v2?

[2ZZ]

Has there been any progress on this? I am blocked by this issue too

[cxhansen]

Hi. We would love to provide an updated Istio integration that doesn't rely on Mixer, but we probably won't be able to do so until sometime in autumn at the earliest. In the meantime, you can continue to use this adapter or use our Telemetry SDK to send metrics and traces to New Relic. /cc @jfjoly

[DanielPanjaitan]

Hello, this is my first time using new relic. Can this adapter be used in Istio version 1.5.0? Or do we have to wait for the latest release of this adapter to be used in that version? Because we have deployed adapter version 2.0.2 on istio 1.5.0, there are no errors, but no metrics and spans found in new relic.

newrelic-istio-adapter 2.0.2 istio 1.5.0

[cxhansen]

@DanielPanjaitan, it looks like you can re-enable the mixer in Istio 1.5 (and 1.6).

[DanielPanjaitan]

@DanielPanjaitan, it looks like you can re-enable the mixer in Istio 1.5 (and 1.6).

Thanks @cxhansen. Hope new release with support for non-mixer related telemetry will come soon.

[mhaddon]

If you want to run without mixer, it should be possible for Istio to report traces to newrelic "directly", and completely bypass this service.

I have tested this and this works, but the data may not be formatted ideally so more work might be required. So maybe an intermediate service is required to transform and annotate the data.

---

Istio cannot send its traces directly to newrelic because newrelic requires authentication to be passed with each request. (Logically). If you tell Istio to send its traces to a path with a query-string it will bork out (it only accepts host:port format).

However, we can set up a reverse nginx proxy setup to manage this for us.

---

We can create a simple nginx mirror proxy as such:

server {
    listen 8080 default_server;

    error_log  /dev/stdout warn;
    access_log  /dev/stdout;
    log_subrequest on;

    location = /newrelic {
        internal;

        resolver 1.1.1.1;

        proxy_ssl_server_name on;
        proxy_ssl_verify off;
        proxy_http_version 1.1;
        proxy_read_timeout 60s;

        proxy_set_header Host localhost;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_pass https://trace-api.newrelic.com/trace/v1?Api-Key=${NEW_RELIC_API_KEY}&Data-Format=zipkin&Data-Format-Version=2;
    }

    location ~ ^/(.*) {
        mirror /newrelic;

        resolver 127.0.0.1:53 ipv6=off;

        proxy_ssl_server_name on;
        proxy_ssl_verify off;
        proxy_http_version 1.1;
        proxy_read_timeout 60s;

        proxy_set_header Host localhost;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_pass http://zipkin.yourjaegernamespace.svc.cluster.local:9411/$1$is_args$args;
    }
}

The 127.0.0.1:53 resolver is just a simple resolver that resolves internal kubernetes dns names for the proxy_pass.

If you use the templating feature of the nginx docker: https://hub.docker.com/_/nginx, then we can replace the NEW_RELIC_API_KEY with an environment defined by a secret in kubernetes that contains our API_KEY.

Note, if you are using the eu newrelic datacenters than the url is trace-api.eu.newrelic.com.

---

If you make this mirror proxy (you can scale it out). You can just simply point everything that wants to send traces to it, and the data will be mirrored to newrelic.

---

To configure Istio you want to set spec/values/global/tracer/zipkin/address in IstioOperator to point to your new mirror proxy. Or in istio-system change the istio configmap under: defaultConfig/tracing/zipkin/address to point to your new mirror proxy. For me: "newrelic-tracer-proxy.observability.svc.cluster.local:9411"

---

You want both requests to return a 202 http status code.

I also recommend that you add more verbose log lines and log sampling. Just ensure you do not log the API_KEY.

---

Furthermore, you should be able to ratelimit the mirror to newrelic (https://www.nginx.com/blog/rate-limiting-nginx/). So you can keep a more complete set of data within the cluster, while not exhausting your limits with newrelic.

[dinup24]

We explored a couple of options to export Istio (v1.5.2) metrics to New Relic. The details of the two approaches are decsirbed here - https://medium.com/@dinup24/export-istio-metrics-to-new-relic-7bb3d6820cc3

We eventually adopted Option 2

[olluch]

Istio 1.6 is end of Support. If we follow the same trend, support for Istio 1.7 will end in about 3 months and Istio 1.8 does not support mixer any longer. Do you have any plans to release support for telemetry v2 within the next 3 months?

[iekdosha]

Hi. We would love to provide an updated Istio integration that doesn't rely on Mixer, but we probably won't be able to do so until sometime in autumn at the earliest. In the meantime, you can continue to use this adapter or use our Telemetry SDK to send metrics and traces to New Relic. /cc @jfjoly

Hi, is there any news regarding using telemetry without Mixer?

[jfjoly]

@iekdosha have you looked at the post mentioned in this comment ? Does it work for you and if not can you share the details of the issues of this approach?

[exploremani]

I am having a similar issue as well. Have installed the istio adapter 2.0.3 on my istio cluster running with 1.7.3. There is no error reported in the POD logs. but not seeing any metrics in the newrelic end. not sure how to trouble shoot even. Is there a different approach to be done for the istio adapter deployment?

[krishna295]

same here, no error on istio adaptor, is this because Mixer is not avaialable?

[exploremani]

@krishna295 - Yes, its the Telemetry the istio adapter is depending on to send the metrics. After enabling the Telemetry I can see the metrics in New relic. There are 2 ways I could do the Metrics to newrelic, one through the istio adapter and another one through the Prometheus integrating to newrelic. My main issue now seems to be the distributed tracing. I couldn't get the distributed tracing working in version istio1.7. As the Telemetry has been depreciated since istio1.5 and completely removed from istio1.8. Not really having much answer around for the issue. To me it appears istio-adpater is not the way to go after istio1.8