canonical / microcloud

Automated private cloud based on LXD, Ceph and OVN
https://microcloud.is
GNU Affero General Public License v3.0
260 stars 36 forks source link

github: run system-tests on larger runners (8cores+32G) #334

Open simondeziel opened 1 week ago

simondeziel commented 1 week ago

Despite the additional HW resources, the test run-times didn't improved drastically but hopefully this should be more robust. This should also open the door to doing heavier tests without running into space or RAM limits. The CPU overcommit ratio is also better thus reducing the steal time.

Test run-times can be compared with https://github.com/canonical/microcloud/actions/runs/9783672548/job/27012801592

Note: the System (1.22.x, interactive) still takes ~3 hours and sometimes fail, same as with the regular runners :/

simondeziel commented 3 days ago

Things are way more flaky on 24.04 it seems. The wrong/unexpected disk is often picked causing multiple failures. I'm a bit unsure how to improve that so if anyone has an idea, please let me know.

simondeziel commented 2 days ago

A new failure reason:

10.223.175.0/24
                                                 # workaround for set -e
'
+ sed '/^\s*#/d; s/\s*#.*//; /^$/d'
Error: Failed to bootstrap local MicroCeph: Failed to run post-bootstrap actions: Failed to start monitor: Failed to run: snapctl start microceph.mon --enable: exit status 1 (error: error running snapctl: snap "microceph" has "auto-refresh" change in progress)
+ cleanup