test_slurm_dumps_stder_to_file flaky locally

[eivindjahren]

Had the following failure happen locally while running pytest tests/unit_tests -n logical -m "not integration_test"

    async def test_slurm_dumps_stderr_to_file(tmp_path, job_name):
        os.chdir(tmp_path)
        driver = SlurmDriver()
        failure_message = "failURE"
        await driver.submit(0, "sh", "-c", f"echo {failure_message} >&2", name=job_name)
        await poll(driver, {0})
        assert Path(f"{job_name}.stderr").exists(), "Slurm system did not write stderr file"

>       assert (
            Path(f"{job_name}.stderr").read_text(encoding="utf-8").strip()
            == failure_message
        )
E       AssertionError: assert '' == 'failURE'
E         - failURE

.../tests/unit_tests/scheduler/test_slurm_driver.py:274: AssertionError

[berland]

Not able to reproduce with pytest tests/unit_tests/scheduler/test_slurm_driver.py -k dumps_stderr -sxv --count 4000 -n 6

[eivindjahren]

@berland Perhaps very unlikely then or only happens when the machine is busy with other things?