Ports getting assigned incorrectly for client-server pairs in uperf workload (Multiple pairs use case)

[vishnuchalla]

Describe the bug

• Port number mismatch between client and server pairs in uperf. This is leading to the failures in the client execution.
• And also in the ClusterIP service when multiple pairs are created there is mismatch in the ports number being assigned leading to issues in packets transfer from client to server.

To Reproduce Steps to reproduce the behavior:

• For without service use case run this CR

  apiVersion: ripsaw.cloudbulldozer.io/v1alpha1
  kind: Benchmark
  metadata:
  name: uperf-benchmark
  namespace: benchmark-operator
  spec:
  clustername: myk8scluster
  elasticsearch:
  url: https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443
  #test_user: username_to_attach_to_metadata
  workload:
  # cleanup: true
  name: uperf
  args:
    serviceip: false
    hostnetwork: false
    networkpolicy: false
    multus:
      enabled: false
    pin: true
    pin_server: "ip-10-0-137-160.us-west-2.compute.internal"
    pin_client: "ip-10-0-160-15.us-west-2.compute.internal"
    samples: 1
    kind: pod
    pair: 10
    test_types:
      - stream
    protos:
      - tcp
    sizes:
      - 16384
    nthrs:
      - 1
    runtime: 30

• For Usecase with a service run this CR.

  apiVersion: ripsaw.cloudbulldozer.io/v1alpha1
  kind: Benchmark
  metadata:
  name: uperf-benchmark
  namespace: benchmark-operator
  spec:
  clustername: myk8scluster
  elasticsearch:
  url: https://search-perfscale-dev-chmf5l4sh66lvxbnadi4bznl3a.us-west-2.es.amazonaws.com:443
  #test_user: username_to_attach_to_metadata
  workload:
  # cleanup: true
  name: uperf
  args:
    serviceip: true
    hostnetwork: false
    networkpolicy: false
    multus:
      enabled: false
    pin: true
    pin_server: "ip-10-0-137-160.us-west-2.compute.internal"
    pin_client: "ip-10-0-160-15.us-west-2.compute.internal"
    samples: 1
    kind: pod
    pair: 5
    test_types:
      - stream
    protos:
      - tcp
    sizes:
      - 16384
    nthrs:
      - 1
    runtime: 30

Expected behavior Client and Server must be communicating on the same port numbers in order for the workload to execute.

Screenshots

• For multiple pairs without a service use case.

  1. 
  2. 
  3. In the above screenshots we have list of client server pods(1), if you consider one of the clients and check for its configuration (2) you will notice that its corresponding server pod is listening to a different port from the one client is trying to communicate. And also even if there is failure in the uperf execution (3) the client status is being marked as completed due to which this issue did not get noticed till now. Both operator and wrapper repos require code changes to bring port numbers inline and mark the correct status of the client pods after execution.

• For multiple pairs with a service use case.

  1. 
  2. 
  3. 
  4. 
  5. In the above screenshots we have a list of resources (1), if you look at any client and server trying to communicate on 30000 port(2) and (3) are running successfully but the rest of them are failing(4) and (5) because those ports are not provided by the service. So we need to bring the port numbers inline here as well

[mukrishn]

Good find, recent CI executions are failing on running more than 1 pair. Airflow logs have executions of 1, 2 & 4 pairs but results are available only for single pair.

[mukrishn]

Airflow task marked that test as completed because of false positive status of the benchmark resource. Client pods gets completed even on 100% packet loss and hence benchmark gets completed successfully as well. Need to fix that in the uperf wrapper script.

[jtaleric]

Airflow task marked that test as completed because of false positive status

Can you elaborate here? In Airflow, is it just ignoring the status code?

Client pods gets completed even on 100% packet loss and hence benchmark gets completed successfully as well.

😨

[mukrishn]

Can you elaborate here? In Airflow, is it just ignoring the status code?

It is not an Airflow problem, we observed the same behavior even without it.
It's because the benchmark wrapper not failing it in this case(screenshot), instead client pod goes to completed state and benchmark resource as well.

[vishnuchalla]

Yes. Even though the client pod has a failure. The benchmark status is marked as complete.