netobserv / flowlogs-pipeline

Transform flow logs into metrics
Apache License 2.0
75 stars 23 forks source link

connection focused flowlogs simulator #372

Closed ronensc closed 1 year ago

ronensc commented 1 year ago

It might be useful to create a simulator that generates flowlogs that form connections with predefined characteristics such as:

Note: The rate and the length could be sampled from some distribution to allow variability.

The simulator may expose a Prometheus endpoint to publish its stats.

This simulator will allow us to test FLP in various scenarios.

Configuration example:

classes:
- srcSubnet: "10.1.1.0/24"
  dstSubnet: "10.1.2.0/24"
  srcPortRange: 20000-30000
  dstPortRange: 80,443
  newConnectionsRate:
    distribution: exponential
    parameters:
      mean: 10s
  timeLength:
    distribution: normal
    parameters:
      mean: 2m
      std: 5s
  dataLength:
    distribution: normal
    parameters:
      mean: 2000000 # 2MB
      std: 1000 # 1KB
- srcSubnet: "10.1.1.0/24"
  dstSubnet: "10.1.3.0/24"
  srcPortRange: 20000-30000
  dstPortRange: 80,443
  newConnectionsRate:
    distribution: exponential
    parameters:
      mean: 20s
  timeLength:
    distribution: normal
    parameters:
      mean: 10m
      std: 10s
  dataLength:
    distribution: normal
    parameters:
      mean: 1000000000 # 1GB
      std: 1000000 # 1MB
eranra commented 1 year ago

@ronensc an alternative might be to use a tool like iPerf with known configuration in something like a Docker environment and capture the real simulated traffic. There are so many network simulators out there that we can find one we can use and do not have to develop on our own.

BTW: If we use such a tool and add the eBPF agent into the story, we will test an even broader end-to-end scenario that also tests the eBPF code. @jotak, is there anything from the past and/or from the other teams under networking in RH that can be used for that >?