Add functional tests for grub lib

[rgildein]

• add functional tests for grub lib
  • the feature tested for grub is required to run on a VM instead of a container, so I changed the command in tox to use a VM with LXD
• add _update_config function to update charm config file instead of overwriting it
• fix writing lines in _save_config function
• raise ApplyError if grub-mkconfig failed, since it is failing if wrong configuration was provided

[rgildein]

@benhoyt Can me moved CI to runner, which will be able to run VMs with lxd? It's mandatory to run integration tests for grub on non container machine, or potentially we can mocked that function.

[benhoyt]

@rgildein Approved now. I'm not quite sure what you mean by "Can me moved CI to runner" -- can you clarify and/or propose that in the GitHub workflow YAML?

[rgildein]

@rgildein Approved now. I'm not quite sure what you mean by "Can me moved CI to runner" -- can you clarify and/or propose that in the GitHub workflow YAML?

If you check this line from logs form Integrations tests, it is saying that on that runner we could not create VM machine. I though that we are using self hosted runner on which is not possible to do this, but apparently this is an issue of GitHub runner where nested virtualization is disabled. I'll check how we can enable it.

[benhoyt]

@rgildein Per your Mattermost message, you say:

The issue is that GitHub runners do not allow nested virtualization and the gtub lib require to be run on VM and not container. Options:

• We can run only Ubuntu integration tests directly on the runner VM with run: tox -e integration-ubuntu. There is no option to use CentOS. e.g. using runs-on: centos-latest
• We can run CI on self-hosted machine, which can use lxc launch --vm ....
• We can run CI on larger GitHub runner 1, where it's possible to enable KVM. However it's billed.
• We do skip for those functional tests if they are run on container machine for now and test it only manually.

I think we should do option 1 for now -- run Ubuntu integration tests directly on a VM runner. Fairly simple, and gets us at least some automated testing for this.

[rgildein]

The linting error was fixed in #105.

[benhoyt]

The code looks good. Looks like the Ubuntu integration tests are failing / timing out, though.

[rgildein]

The code looks good. Looks like the Ubuntu integration tests are failing / timing out, though.

The tests are not failing, but the "Setup operator environment" is failing. I think it's due some change in system config. We can check all tests and properly clean any configuration or simply run juju-systemd-notices first or even better move that to separate job. What do you think? To me the best option is to move it to separate job.

[benhoyt]

For the record, @NucciTheBoss said:

I wouldn't be surprised if the grub functional tests modified something that prevented lxd from restarting the containers correctly. For example, if you don't clean up your NFS kernel server running inside an LXD container, lxc/lxd commands will hang for most of eternity unless. That seems to have been what happened with this job https://github.com/canonical/operator-libs-linux/actions/runs/6025011553/job/16344855206

So we should probably get to the bottom of that at some point. In the meantime, I'm going to merge this. We can always improve / isolate CI later.