Flaky Python tests

[rtibbles]

Observed behavior

This is a tracking issue to keep track of a few different Python tests that occasionally fail:

• In kolibri/core/tasks/test/taskrunner/test_job_running.py test_can_cancel_a_job
• In kolibri/core/tasks/test/taskrunner/test_worker.py test_high_tasks_dont_wait_when_regular_workers_busy can sometimes fail with the fetched job being None - this test seems to specifically only fail on MacOS test runners.
• In kolibri/core/content/test/test_content_app.py for Postgres tests only, anything using _assert_nodes - can seemingly fall victim to inconsistent ordering on occasion (not fixed by #8711) - it seems that the ordering did not go far enough, most recent failures suggest that this is actually a result of inconsistent ordering of the files! Fixing that should fix the issue.
• ~In kolibri/core/device/test/test_syncqueue.py for Windows test_position_in_queue can occasionally fail with the position being 4 instead of 5~ - I think this was resolved by adding time.sleep into tests to ensure proper ordering by time.
• ~kolibri/core/test/test_key_urls /api/auth/portal/validate_token/ request is sensitive to Internet connectivity and will give a 500 when unable to connect - this should be updated to give a more semantic error.~ Fixed in #8559
• Import export test appears to actually be trying to connect to Studio? Run with no Internet connection to try to replicate: kolibri/core/content/test/test_import_export.py

[vkWeb]

Will look into the first one!

[rtibbles]

Will look into the first one!

I suspect, but am not certain, that it is happening during the multiprocessing run of the tests.

[vkWeb]

I suspect, but am not certain, that it is happening during the multiprocessing run of the tests.

Will need to look into why that's the case.

[marcellamaki]

Closed in error -- only partially fixed

[dbnicholson]

I ran into the last one and it comes from a lack of consistent request mocking. That was exacerbated by 8e64e45b8769a6c7fb172a30f55192e3be2a7a37 since it now does a HEAD request before any GET requests. I've used requests-mock in another project and it makes it a lot harder to screw up since you'll immediately get errors for any URLs that haven't been registered for mocking.

[rtibbles]

More details for the MacOS worker test failure:

_____ TestWorker.test_high_tasks_dont_wait_when_regular_workers_busy[True] _____

self = <kolibri.core.tasks.test.taskrunner.test_worker.TestWorker object at 0x11d78cc70>
worker = <kolibri.core.tasks.worker.Worker object at 0x119d76560>

    def test_high_tasks_dont_wait_when_regular_workers_busy(self, worker):
        # We have one task running right now.
        worker.future_job_mapping = {"job_id": "future"}

        job = Job(id, args=(10,))
        worker.storage.enqueue_job(job, QUEUE, Priority.HIGH)

        job = worker.get_next_job()
        worker.future_job_mapping.clear()

        # Worker must get this job since its a 'high' priority job.
>       assert isinstance(job, Job) is True
E       assert False is True
E        +  where False = isinstance(None, Job)

kolibri/core/tasks/test/taskrunner/test_worker.py:202: AssertionError