search

kubermatic / machine-controller

Apache License 2.0
307 stars 130 forks source link

Reworking E2E tests for Machine Controller to support external CCM #1626

Open ahmedwaleedmalik opened 1 year ago

ahmedwaleedmalik commented 1 year ago

Kubernetes maintainers have long been discouraging the usage of In-tree cloud controller manager(CCM) in favor of external CCM. In-tree providers are planned to be removed from the Kubernetes core and we are already affected with:

For now, we are ignoring or skipping e2e tests for the affected versions but this is not ideal/acceptable, it's important to ensure that the e2e tests are reliable and up-to-date.

Proposed solution

Switch from KIND to KKP for machine-controller E2E tests.

  1. Create a dedicated KKP setup(or dedicated seed) for machine-controller. Connectivity for nodes would work out of the box and we won't need any hacks for CCM or CNI.
  2. Creating user clusters for e2e tests. We have multiple options for that; we should evaluate and choose:
    1. Long-standing user clusters per cloud provider with no machine deployments. MC will create the required machine deployments for tests and then clean them up.
    2. When running tests, create a user cluster per cloud provider. We can check if a user cluster for the provider already exists and then just use it.
    3. Dedicated user cluster per CI job; this is not ideal due to resource constraints and the cost incurred.
  3. Run our test suite to check that the machine came up and was ready. We don't need conformance tests etc. here.

This would help solve https://github.com/kubermatic/machine-controller/issues/1462 as well.

NOTE: Proper cleanup jobs should be in place right from the start to avoid broken/cancelled jobs leading to orphaned resources

Alternate solution

embik commented 1 year ago

Will also fix #1620.

kubermatic-bot commented 1 year ago

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

ahmedwaleedmalik commented 1 year ago

/remove-lifecycle stale

ahmedwaleedmalik commented 1 year ago

With https://github.com/kubermatic/machine-controller/pull/1709, we are dropping support for Kubernetes 1.25 and that makes the e2e tests for OpenStack no-op. Since they require external CCM.

ahmedwaleedmalik commented 10 months ago

In-tree providers have been disabled by default via https://github.com/kubernetes/kubernetes/pull/117503.

kubermatic-bot commented 7 months ago

Issues go stale after 90d of inactivity. After a furter 30 days, they will turn rotten. Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

embik commented 7 months ago

/remove-lifecycle stale