Support diverse tracing backends in Mixer

[ZackButcher]

We have no story around pushing traces to various backends, and how new tracing implementations fit into Istio. We need to develop a strategy and produce designs around ingesting trace data and what we expect Mixer to do with it. This is complicated by the fact that there is no standard wire format for traces.

[ZackButcher]

For context, today our story is:

• Envoy implements application tracing via Zipkin
• Mixer implements application tracing via opentracing, and uses the Zipkin opentracing implementation for interoperability with Envoy.

In both cases the Istio components push their traces directly to a Zipkin collector (i.e. it does not flow through the Istio telemtry pipeline). There is no opportunity for an operator to touch the trace data before it's ingested by the collector.

---

In the near future (0.2.x release time frame) we plan to expose a faux Zipkin collector API as an adapter, and Envoy will send its traces to Mixer. That will provide a hook for others to extend trace processing in Mixer, but inputs will be Zipkin specific.

Long term, if the project to develop a standard wire format succeeds, we plan to ingest trace data conforming to that format. That format will provide a common starting point for adapter authors to work from: their business logic can accept that common format, transform it to their own wire format, and augment the traces with values derived from attributes.

[geeknoid]

What I'd like Mixer to do here is (and this applies beyond tracing)

• Implement well-known/popular telemetry APIs. So have Mixer implement the OpenTracing API and the OpenMetrics API (when it appears)

• Provide a generic framework in Mixer which converts the incoming telemetry API calls into Mixer instances.

The idea is that you can redirect the output of existing services to Mixer, and then apply all Mixer policy and routing machinery as normal. So basically, we can capture service telemetry sans making changes to service code, while still providing all the flexibility that the platform enables.

[ZackButcher]

I agree in principle, but as an implementation detail I'd prefer to see us expose these APIs as services external to Mixer that collect their data then call the Mixer's disintermediated API. The operator can then choose to run these services side-by-side with Mixer, or deployed and scaled separately. I'm in the process of writing up a doc proposing that pattern as the best practice for API extensibility in Mixer. I should be sending it out by the end of the week.

[geeknoid]

Agree with 100% with the direction. Whether Mixer implements these "telemetry transcoders" itself or via an external agent is an implementation detail.

On Wed, Jul 19, 2017 at 4:28 PM, Zack notifications@github.com wrote:

I agree in principle, but as an implementation detail I'd prefer to see us expose these APIs as services external to Mixer that collect their data then call the Mixer's disintermediated API. The operator can then choose to run these services side-by-side with Mixer, or deployed and scaled separately. I'm in the process of writing up a doc proposing that pattern as the best practice for API extensibility in Mixer. I should be sending it out by the end of the week.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/istio/mixer/issues/797#issuecomment-316549988, or mute the thread https://github.com/notifications/unsubscribe-auth/AVucHWdAHKpmhxuwnA3MgMLvxulXlVYgks5sPpEWgaJpZM4Nxu3P .

[douglas-reid]

I think the idea of doing telemetry transcoding is a bit ambitious, and not strictly within Mixer's wheelhouse. We should carefully consider existing models, like collectd and fluentd, and whether or not it makes sense to have Mixer involved in the transcoding (to Mixer instances) at all. Writing a Mixer plugin for collected / fluentd may make more sense than attempting any transcoding ourselves.

The value proposition for places that already use such tools for collection and reporting of telemetry of going through yet another piece of infrastructure will need to be very clear (as compared to say having galley+pilot config/deploying collectd agents).

I also disagree with the premise that we have no story around pushing traces to diverse tracing backends. While we don't have a named traces mesh function, you can generate traces using the access-logs aspect -- and this can be done for a diverse set of backends. What you can't do is ingest trace data. But you can't do that for any other kind of data either.

Ingestion and modification of trace data will be considerably harder to do than metrics and logs... but unlike logs, at least there is consensus forming around tracing formats.