K8SSAND-1449 ⁃ Consider Traefik as a prerequisite for e2e tests

[adutra]

Traefik is currently deployed as part of the beforeTest steps for each e2e test that requires ingresses.

Several things are wrong with this approach:

• Multiple installs: one single install of Traefik should be enough for all tests.
• Also, Treafik should be considered a prerequisite for tests and be installed only once, prior to the e2e test execution. This should save us some time when running multiple e2e tests locally.
• The tests themselves don't need to know that Traefik is being used: the only thing they need is a URL reachable from outside the cluster. Traefik should be an implementation detail and should be replaceable by any other ingress framework.
• Sticking to Traefik won't play well if/when our tests will be run against cloud providers that favor other ingress providers, see #466.

I suggest the following:

• Consider the deployment of an Ingress controller a pre-requisite
• Create make targets to install Traefik locally with Helm, call them when setting up a test cluster.
  • Traefik should be configured to watch all namespaces.
  • For local clusters, the current Helm values file can be reused.
• Remove all the code in e2e framework to deploy Traefik programmatically.
• We still need to create the ingresses as part of the e2e test, since the namespace and service names are specific to the test. For that:
  • Create new test flags: stargateIngressUrl and reaperIngressUrl with the following defaults for local clusters: stargate.127.0.0.1.nip.io and reaper.127.0.0.1.nip.io.
  • Then the e2e framework will generate 2 ingress objects based on these templates:
  • Stargate
  • Reaper
  • The namespace and service name will be kustomized to match the e2e test current namespace + service name (as it is done already); the host field will be kustomized to include the flag values (new).
  • The templates above are specific to Traefik; we will need to use another ingress type. The most generic option is to use Kubernetes Ingress, but it doesn't handle TCP very well. We will need to investigate how to proceed with testing the CQL port, but one option would be to disable CQL port testing through a flag, if the underlying Ingress controller doesn't handle it.

┆Issue is synchronized with this Jira Task by Unito ┆friendlyId: K8SSAND-1449 ┆priority: Medium

[adutra]

I'm close to having a working PR. I suggest however to change the goal slightly and still consider Traefik a prerequisite, but keep it as our de facto ingress controller.

I'm leaning towards this for two reasons:

• Traefik handles TCP traffic and is nice for testing the CQL protocol;
• Traefik can be very easily installed on cloud providers – I just successfully installed it on GKE with no issues.

By making Traefik the de facto ingress controller, the e2e tests can more easily create ingress routes, since the same ingress definition will work regardless of whether the test is being executed locally, or on a cloud provider.