Functional tests sometimes fail

[sudeephb]

Sometimes(Especially when running as part of Release to Edge workflow), the functional tests fail. The following error occurs:

=================================== FAILURES ===================================
_______________________ test_default_config[jammy-local] _______________________

app = <Application entity_id="sysconfig-jammy-local">
jujutools = <juju_tools.JujuTools object at 0x7fd34c1e2020>

    async def test_default_config(app, jujutools):
        """Test default configuration for grub, systemd and cpufrequtils."""
        unit = app.units[0]
        not_expected_contents_grub = [
            "isolcpus",
            "hugepages",
            "hugepagesz",
            "raid",
            "pti=off",
            "intel_iommu",
            "tsx=on",
            "GRUB_DEFAULT",
            "default_hugepagesz",
        ]
        grub_path = "/etc/default/grub.d/90-sysconfig.cfg"
        RETRY(
>           await jujutools.check_file_contents(
                grub_path, unit, not_expected_contents_grub, assert_in=False
            )
        )

tests/functional/test_deploy.py:77: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/functional/juju_tools.py:98: in check_file_contents
    content = await self.file_contents(path, target)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <juju_tools.JujuTools object at 0x7fd34c1e2020>
path = '/etc/default/grub.d/90-sysconfig.cfg'
target = <Unit entity_id="sysconfig-jammy-local/0">

    async def file_contents(self, path, target):
        """Return the contents of a file.

        :param path: File path
        :param target: Unit object or unit name string
        """
        cmd = "cat {}".format(path)
        result = await self.run_command(cmd, target)
>       return result["stdout"]
E       KeyError: 'stdout'

tests/functional/juju_tools.py:86: KeyError
=========================== short test summary info ============================

It looks like a race condition since it doesn't occur everytime. This needs to be fixed and made more predictable.

[aieri]

not observed recently, and we've changed how the release workflow works. I'll close for now, we can reopen if it happens again