search

netobserv / flowlogs-pipeline

Transform flow logs into metrics
Apache License 2.0
69 stars 21 forks source link

Create new transformer into OpenTelemetry semantics #311

Closed eranra closed 4 months ago

eranra commented 1 year ago

Focus on Flow-Logs / Connection-Logs first Metrics as a second phase

eranra commented 1 year ago

ref: https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod

jotak commented 1 year ago

I'm not sure if it needs to be an exporter, or a transformer. Could it be just a transformer, that does the semantic fields naming and adaptation, and then it can be plugged with prometheus/loki without any change on the exporter side?

jotak commented 1 year ago

See also the conventions for network attributes: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/span-general/#network-transport-attributes But this is where the spec is limited and not a super good fit for us: there's either k8s semantic conventions for the observation point, or network semantic conventions for in/out communications, but these domains cannot be composed; e.g. there's no semantic convention for a "destination pod" as far as I know (something like peer.k8s.pod.name doesn't exist in spec)

bdarfler commented 1 year ago

Would this also include exporting in OpenTelemetry Line Protocol? It seems semantics and protocol support would be best served together.

eranra commented 1 year ago

Would this also include exporting in OpenTelemetry Line Protocol? It seems semantics and protocol support would be best served together.

Hi @bdarfler. I agree on the need for both semantics and protocol to be supported --- this is still not defined to the end, and we are soon going to start working on that ( @KalmanMeth ) --- can you share more details/motivation for the protocol? Consumption by which component?

bdarfler commented 1 year ago

I could see users being interested in sending this data to the OpenTelemetry Collector and then sending it to their vendor or backend of choice.

The pipeline architecture of flowlogs and the OTel Collector is quite similar so there may also be some benefit for flowlogs to interoperate with the OTel Collector and leverage its processing and transformation capabilities rather than reimplement all that within flowlogs.

I can see an argument for keeping high-value protocol support like loki and prometheus in flowlogs but for the longer tail (like paid vendors, etc.) it might be nice to offload that support to the existing support in the OTel Collector

eranra commented 1 year ago

@bdarfler ok, got it; THX for the explanation--- it does make sense; although the Network Observability pipeline includes various "network specific" and "kube specific" capabilities that are required + it also generates metrics from the logs --- I did not follow up with OTel capabilities in this space. Still, I would not be surprised if there is some overlap.

KalmanMeth commented 4 months ago

Issue is addresed by #531 and #577.