search

kubernetes / autoscaler

Autoscaling components for Kubernetes
Apache License 2.0
8.04k stars 3.96k forks source link

spike: e2e smoke test over a CAPI ephemeral cluster #2942

Closed enxebre closed 1 year ago

enxebre commented 4 years ago

As a dev I'd like to have some smoke tests to run on PR to validate I'm not breaking e2e expectations. /area provider/cluster-api

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

detiber commented 4 years ago

/remove-lifecycle stale

ncdc commented 4 years ago

I'm interested in seeing this happen - can we brainstorm & enumerate things that would be needed for this to be realized?

enxebre commented 4 years ago

That's great @ncdc, we are planning on doing efforts to make this happen as well.

WRT your points above, I think we need to answer what would it take for us to run this suite https://github.com/kubernetes/kubernetes/tree/master/test/e2e/autoscaling against the capi provider in GCP/AWS/Azure? TODO: enumerate the technical impediments. TODO: enumerate cloud infra costs impediments.

If we can overcome the TODOs above we could run periodic jobs for the CAPI provider against those clouds as in https://testgrid.k8s.io/sig-autoscaling-cluster-autoscaler#Summary&show-stale-tests=. That suite seems take ~4h https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-autoscaling/1291216872313720836

Since the test suite above takes ~4h we'll need a different smoke test suite for PRs. Ideally we'd run this against any CAPI supported provider e.g AWS/Azure/GCP and also against a fast ephemeral CAPI environment e.g libvirt/docker/kubemark. To accelerate the job we could use intentionally reduced timeouts for things like scale-down-delay-after-add, scale-down-delay-after-delete, scale-down-delay-after-failure, scale-down-unneeded-time... This is fine, it's a smoke test and real values would be use in the periodic jobs.

If there's hard impediments to run against the clouds on PRs we could start by only running the job against libvirt/docker/kubemark. In terms of what tests we want to run I'd start simple as we have nothing today: Get the plumbing in place. Run a trivial smoke test suite e.g expose machineSets to autoscaler, run targeted workload watch it scale out, delete workload, watch it scale down. Then we go from there.

detiber commented 4 years ago

I don't necessarily think that cost is an impediment wrt cloud infra, but there are likely several other cloud infra related impediments:

We could potentially look at alternatives to boskos-managed accounts, but I think I personally would prefer that route, since we can have a janitor process ensure cleanup of resources if tests fail or are otherwise abandoned before cleaning up after themselves and wouldn't require a separate process to try and mark/sweep resources.

All that said, while it would be nice to have testing against actual cloud providers, I'm not sure it's necessarily needed. Since the interaction is through Cluster API core resources, I think we'd get sufficient signal using any provider. CAPD is probably a good initial target, since it can run in prow, but it could potentially run into resource limits at higher scale and be susceptible to noisy neighbor flakiness.

@ncdc @vincepri we've talked about potentially having a "mock" or "fake" provider in the past, this might be a good use case for that. envtest from controller-runtime might make a decent base for building such a provider.

elmiko commented 4 years ago

WRT your points above, I think we need to answer what would it take for us to run this suite https://github.com/kubernetes/kubernetes/tree/master/test/e2e/autoscaling against the capi provider in GCP/AWS/Azure? TODO: enumerate the technical impediments. TODO: enumerate cloud infra costs impediments.

i've spent several weeks going over these tests and imo i think we need to refactor them to be more provider agnostic. currently they have a heavy reliance on gcp/gke specific infrastructure. i would like to see us take the core of these tests and make the provider implementation more generic.

i think if we can convert those tests to use a more generic provider, with a clean abstracted interface, we will make it easier for provider-specific teams to add their tests (i am talking about autoscaler in-tree providers here). for the CAPI specific tests then, we would just need to create our provider implementation and then we could leverage all the clouds covered by CAPI.

so far, i have started breaking down the e2e tests in a separate repo. my intent is to create a library which can be consumed by providers to create testing suites. my thinking is that if the tests can be contained into something that is easily usable by the larger community, it will be easier to migrate those changes if the need arise (eg if the community wants to have them centralized here or in k/k). sadly, i don't have much to share yet i am still working around some issues with configuring the external package and breaking some hard dependencies on k/k.

elmiko commented 4 years ago

@ncdc @vincepri we've talked about potentially having a "mock" or "fake" provider in the past, this might be a good use case for that. envtest from controller-runtime might make a decent base for building such a provider.

i think this is an interesting idea. it shouldn't be too difficult to build a mock which could present the CAPI objects back to the autoscaler. i do wonder about the node and pod detection pieces though.

ncdc commented 4 years ago

cc @benmoss

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

elmiko commented 3 years ago

this issue is going to take time to properly address. we have had some good discussions at the sig autoscaler and sig testing meetings and i believe we have a plan to move forward. i am still working on a KEP to describe the overall action, and @benmoss has proposed #3651 , which adds the framework we will use for e2e testing. there are still some details to finalize in terms of how and where the tests would run and what type of default configuration would be used. /remove-lifecycle rotten

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

elmiko commented 3 years ago

i am still interested in seeing this happen, the work has been slow going. /remove-lifecycle stale

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

elmiko commented 3 years ago

i have been continuing to hack on this locally. recently there has been some interesting movement in the cluster-api project around using kubemark for testing in e2e (see https://github.com/kubernetes-sigs/cluster-api/pull/4566).

my current progress is around building a capi cluster with the hollow nodes and then attempting to run e2e tests for the autoscaler on top of that. i think the kubemark approach will bear more fruit than the basic capd approach mainly due to the way that the node resources (cpu, mem, etc) are propagated. in capd, the way docker handles these limits does not apply into the node, wheres it appears that kubemark is doing this properly. having these resources be accurately represented is key to testing the autoscaler in this topology.

/remove-lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

elmiko commented 3 years ago

i am still keenly interested in seeing this move forward. i have had several configuration issues that are blocking me from getting the kubemark cluster (mentioned in my previous comment) working. regardless, i continue to push on this. i have a feeling that once we merge the kubemark testing changes into capi, then we can start to setup automation to try out the autoscaler.

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

elmiko commented 2 years ago

i am working towards making this a reality but it is taking some time to get there. we now have a published kubemark provider for cluster-api and i am adding a feature which will allow us to specify the node resources. once that patch is in the kubemark provider, we should be able to start creating more meaningful tests with the autoscaler and cluster-api.

no estimate on time frame for landing all of this, but i am working towards pushing several pieces forward. will update as progress is made.

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

elmiko commented 2 years ago

slow progress is continuing on this. we have released an updated version of the kubemark provider with support for custom resource definitions. This update also supports scale from zero for the provider. The next steps are to update the clusterapi provider for the autoscaler to support the new features.

still no estimate for completion, but the next steps will involve transforming the test work done by @benmoss to run on top of the capi provider with kubemark enabled.

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

elmiko commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

elmiko commented 2 years ago

i am still working towards this, although it does have a glacial pace ;)

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

elmiko commented 1 year ago

/remove-lifecycle rotten

i'm not sure if we are making progress towards having this smoke test in the autoscaler, although the topic has come up at sig meetings and i believe there might be some support for doing it. i would like to share some work that is happening on the capi kubemark provider, this PR https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/pull/69 proposes a test suite for the kubemark provider that uses the autoscaler for testing. i think it will be important work for creating a repeatable pattern that could be used with tests in this repository.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/autoscaler/issues/2942#issuecomment-1585209409): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.