search

kubernetes-sigs / cluster-api-provider-openstack

Cluster API implementation for OpenStack
https://cluster-api-openstack.sigs.k8s.io/
Apache License 2.0
286 stars 252 forks source link

✨ CI Improvements tracker :page_with_curl: #1814

Open EmilienM opened 8 months ago

EmilienM commented 8 months ago

This list the items that we want to fix or improve in our CI.

Bump Ubuntu & OpenStack versions

Improve Logging

There are a lot of red herring that can confuse the developers when reading CI logs.

Simplify how devstack is configured and run to deploy OpenStack

Right now we use custom shell scripts to configure and run Devstack, which we have to maintain based on OpenStack versions, etc. It'll be nice if we could just consume how the OpenStack community does in upstream CI so we reduce the cost of maintenance.

Artifact gathering

lentzi90 commented 8 months ago

I have been dreaming about a way to reduce the time it takes to set up the devstack. When working locally I create the devstack once and then reuse it for multiple tests to reduce the waiting time, but the CI does it from scratch every time.

I found out that at some point we used to have ready made devstack images and I imagine that the idea was to make the setup faster. Unfortunately I have not had much luck snapshotting a devstack and then start a new from the snapshot though... If we could find a way to do that though, it would be awesome!

EmilienM commented 8 months ago

I have been dreaming about a way to reduce the time it takes to set up the devstack. When working locally I create the devstack once and then reuse it for multiple tests to reduce the waiting time, but the CI does it from scratch every time.

I found out that at some point we used to have ready made devstack images and I imagine that the idea was to make the setup faster. Unfortunately I have not had much luck snapshotting a devstack and then start a new from the snapshot though... If we could find a way to do that though, it would be awesome!

I've added it to the list. @mdbooth had the same idea!

dulek commented 8 months ago

We could think about converging on the CPO and CAPO CIs. Currently they use totally different way of setting up everything.

I'd also love to set up DevStack using upstream Ansible playbooks: https://opendev.org/openstack/devstack/src/branch/master/playbooks/devstack.yaml

EmilienM commented 8 months ago

We could think about converging on the CPO and CAPO CIs. Currently they use totally different way of setting up everything.

I'd also love to set up DevStack using upstream Ansible playbooks: https://opendev.org/openstack/devstack/src/branch/master/playbooks/devstack.yaml

Yes! Definitely, I was about to reach you on that topic :)

mdbooth commented 8 months ago

@EmilienM I think a bunch of folks will buy you many beers if you fix this.

mandre commented 8 months ago

In addition to this great list, the CI also sometimes fails to provision nodes due to insufficient resource available on the hypervisor. When that happens, the node are in ERROR state and CAPO stops reconciling the machine.

Since we apparently can't make the hypervisor bigger, so we should find a way to mark the machine heavy tests exclusive one to another. Alternatively we may look into machine healthchecks so that the failed nodes can be reprovisioned.

mandre commented 8 months ago

I've fixed the gcloud errors with https://github.com/kubernetes-sigs/cluster-api-provider-openstack/pull/1804/commits/e71d3aeac9ddfe8c3068343fa4ff9341aaf3849b, which I might split in its own separate PR.

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

EmilienM commented 4 months ago

/remove-lifecycle rotten

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten