netobserv / flowlogs-pipeline

Transform flow logs into metrics
Apache License 2.0
75 stars 23 forks source link

Allow more generic use of the IPFIX exporter #632

Open jotak opened 6 months ago

jotak commented 6 months ago

Currently the IPFIX exporter in FLP is very tied to the Agent/protobuf input and could crash when used in a different context, due to the assumptions done when casting elements from GenericMap.

It is also tied to the configuration defined in the operator, for instance assuming k8s fields are prefixed with "SrcK8S" or "DstK8S", which shouldn't be hard-coded.

It should be made more generic.


Here's an example that today doesn't work:

  1. Start a first FLP that reads from IPFIX and write to stdout, using this stdout.yaml:
pipeline:
  - name: ingest
  - name: write
    follows: ingest
parameters:
  - name: ingest
    ingest:
      type: collector
      collector:
        hostName: 0.0.0.0
        port: 2057
  - name: write
    write:
      type: stdout
./flowlogs-pipeline --config ./stdout.yaml
  1. Start a second FLP that reads from (agent) IPFIX and rewrite as IPFIX, using this ipfix-relay.yaml
pipeline:
  - name: ingest
  - name: write
    follows: ingest
parameters:
  - name: ingest
    ingest:
      type: collector
      collector:
        hostName: 0.0.0.0
        port: 2055
  - name: write
    write:
      type: ipfix
      ipfix:
        targetHost: 0.0.0.0
        targetPort: 2057
        transport: udp
        enterpriseID: 0
metricsSettings:
  port: 9091
./flowlogs-pipeline --config ./ipfix-relay.yaml --health.port 8081
  1. Start the agent:
export EXPORT="ipfix+udp"
export FLOWS_TARGET_HOST="0.0.0.0"
export FLOWS_TARGET_PORT="2055"
sudo -E bin/netobserv-ebpf-agent

=>

You'll get errors like:

panic: interface conversion: interface {} is uint64, not int64

goroutine 84 [running]:
github.com/netobserv/flowlogs-pipeline/pkg/pipeline/write.setStandardIEValue(0xc00043a800?, 0xc00035a460?)
    /w/flowlogs-pipeline/pkg/pipeline/write/write_ipfix.go:325 +0x1054
github.com/netobserv/flowlogs-pipeline/pkg/pipeline/write.setEntities(0x7f74a0af5400?, 0x0, 0x0?)
    /w/flowlogs-pipeline/pkg/pipeline/write/write_ipfix.go:403 +0x74
github.com/netobserv/flowlogs-pipeline/pkg/pipeline/write.(*writeIpfix).sendDataRecord(0xc000125380, 0xc0003c0090?, 0x0)
    /w/flowlogs-pipeline/pkg/pipeline/write/write_ipfix.go:424 +0xfa
github.com/netobserv/flowlogs-pipeline/pkg/pipeline/write.(*writeIpfix).Write(0xc00065ceb0?, 0x494a0f?)
    /w/flowlogs-pipeline/pkg/pipeline/write/write_ipfix.go:460 +0x20d

.. because of the assumptions about types in GenericMap