search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
189 stars 403 forks source link

EnvoyProxy Integration #9383

Open SubhrataK opened 4 months ago

SubhrataK commented 4 months ago

This issue is to migrate the EnvoyProxy module from beats to agent-based integration.

tehbooom commented 3 months ago

Started development of this integration. For metrics will be following a similar model as the Airflow package.

PR to follow.

tehbooom commented 2 months ago

Metrics currently a blocker per

https://github.com/elastic/beats/issues/39473

ritalwar commented 2 months ago

Just a quick question: I see we already have an EnvoyProxy Beats module, and we just want to create an agent-based integration for it. Why are we planning to use StatsD instead of using the existing EnvoyProxy Beats module for creating this integration?

tehbooom commented 2 months ago

@ritalwar The way our module was collecting the logs was simply doing a request to /stats with maybe 20 or so fields.

Envoy does support Statsd and states it's typically how the metrics are collected.

I did try to use the metric beat module but I noticed I was missing a lot of fields and they were in "Statsd format". So I thought it might be best to simply redo the metrics using Statsd or recreate the module using CEL input.

I think Statsd is the way to forward here as it's what envoy is supporting and other applications are using Statsd as an input like DataDog.

ritalwar commented 2 months ago

If statsd seems like the best way forward based on your analysis, let's go with that, although we can update this schema to include more/missing fields if necessary.

tehbooom commented 2 months ago

I agree that we could update that schema however some of the fields are already histograms like in statsd. I think that would be more work in the long run to basically recreate our statsd module to parse out "statsd like" fields.

ritalwar commented 2 months ago

I think that would be more work in the long run to basically recreate our statsd module to parse out "statsd like" fields.

Agree! let's modify the existing one(statsd) for this integration unless there's a better solution suggested.