Rework hw pytest

[PlagueCZ]

There was a switch to run pytest suite using actual hardware (Mellanox NIC). It was highly specific to a setup with corsslinked ports and could not thus cover everything.

I created a better implementation where the Mellanox NIC is connected to another machine (or another NIC on the same machine with network namespace isolation). Then a special scapy script is run on the remote machine to reflect underlay packets back to the test suite (it needs to mangle some of them to prevent isolation rule from firing).

Currently this needs a directly connected machine with no switch in between, as the MAC addresses used would cause the traffic to be dropped I believe. But this is something I want to address later so this test can be run in other environments.

This is also capable of testing the multiport setup and pf1-proxy functionality (as the pytest suite needs to listen on the proxy port)

---

The change to dpservice C code is actually just a cleanup of a hack to make the previous HW test work.

As for the test suite itself, almost all calls to send() and sniff() were already encapsulated, so adding the de-mangling of packets (for isolation rule prevention) is centralized. There were a few instances of a naked call to send() and sniff() which I encapsulated and it even cleaned the code up.

There is a bit of a messy change to telemetry tests (due to the fact that we change the graph for pf1-proxy, etc.), but this will be fixed once we either remove or rework the way pf1-proxy is implemented.

Apart from the above, the tests themselves are oblivious to this change.

[byteocean]

I do not have the chance to actually run these tests due to lacking of directly connected machines. I get the idea by reading the code, and see the value during the development. What I can comment here is that, preparing an environment, e.g., providing a qcow2 file, setting up VMs and bridges, are currently manual, thus difficult to integrate into CI/CD. It is of course the same issue as the benchmark test.

[guvenc]

@PlagueCZ I am trying to visualize how this test is actually wired including the VFs/VMs, PF0/PF1 (PF1 Tap case ?) and monitoring port (Including the reflector PC) and I can not quite get the complete picture as a mental image. Would it be possible to prepare a really simple draw.io drawing to show how the things are wired/connected and which script is running where. Nothing fancy. Just some boxes and lines with some description text boxes would be fine. We can keep it in the PR and it doesnt have to be part of the codebase.

[PlagueCZ]

Hopefully this image covers the idea properly. You need to know that only "underlay packets" ever reach dpservice's PF due to isolation rules (this is not technically correct, the rules are more complex, but the idea holds). This is what reflector.py uses to direct the packet back to dpservice (leave the packet as-is) or to pytest (change EtherType to 0x1337 so the rules do not match).

Virtual machines are then set-up so they are a transparent bridge (explained in the markdown docs) to simply pass packets between dpservice and pytest.

The schema only shows one PF and one VF for clarity, there is no dependence among any of them, but of course the reflector does perform the same on two interfaces (PF0 and PF1).

[PlagueCZ]

The PF1 tap case is wired the same except that pytest needs to listen on pf1-tap device instead. But that is actually true for everything running on that machine (frr, curl, ping, ...) as PF1 is inaccessible due to the multiport mode.

In that case dpservice is actually doing more, but you can imagine a blackbox bridge between PF1 and the TAP device and it would work the same, it's just that the blackbox is dpservice itself.