mozilla / experimenter

A web application for managing user experiments for Mozilla Firefox.
https://experimenter.services.mozilla.com
Mozilla Public License 2.0
113 stars 175 forks source link

Intermittent Cirrus test failure: test_fetch_schedule_recipes_success #10809

Open jaredlockhart opened 1 month ago

jaredlockhart commented 1 month ago

Seen here: https://app.circleci.com/pipelines/github/mozilla/experimenter/39997/workflows/cef71fb3-6850-4453-91a4-252e37dfb32c/jobs/195683

_____________________ test_fetch_schedule_recipes_success ______________________

self = <MagicMock name='scheduler.add_job' id='140560502149904'>

    def assert_not_called(self):
        """assert that the mock was never called.
        """
        if self.call_count != 0:
            msg = ("Expected '%s' to not have been called. Called %s times.%s"
                   % (self._mock_name or 'mock',
                      self.call_count,
                      self._calls_repr()))
>           raise AssertionError(msg)
E           AssertionError: Expected 'add_job' to not have been called. Called 1 times.
E           Calls: [call(<function fetch_schedule_recipes at 0x7fd6cad98b80>, 'interval', seconds=30, max_instances=1, max_retries=3)].

/usr/local/lib/python3.11/unittest/mock.py:900: AssertionError

During handling of the above exception, another exception occurred:

scheduler_mock = <MagicMock name='scheduler' id='140560502153104'>
remote_setting_mock = <MagicMock name='remote_setting' id='140560502980304'>

    @pytest.mark.asyncio
    async def test_fetch_schedule_recipes_success(scheduler_mock, remote_setting_mock):
        remote_setting_mock.fetch_recipes.return_value = {
            "data": [{"experiment1": True}, {"experiment2": False}]
        }

        await fetch_schedule_recipes()
        remote_setting_mock.fetch_recipes.assert_called_once()

        # Check that no jobs were added to the scheduler
>       scheduler_mock.add_job.assert_not_called()
E       AssertionError: Expected 'add_job' to not have been called. Called 1 times.
E       Calls: [call(<function fetch_schedule_recipes at 0x7fd6cad98b80>, 'interval', seconds=30, max_instances=1, max_retries=3)].
E       
E       pytest introspection follows:
E       
E       Args:
E       assert (<function fe...>, 'interval') == ()
E         Left contains 2 more items, first extra item: <function fetch_schedule_recipes at 0x7fd6cad98b80>
E         Use -v to get more diff
E       Kwargs:
E       assert {'max_instanc...'seconds': 30} == {}
E         Left contains 3 more items:
E         {'max_instances': 1, 'max_retries': 3, 'seconds': 30}
E         Use -v to get more diff

tests/test_main.py:161: AssertionError

┆Issue is synchronized with this Jira Task

jaredlockhart commented 1 week ago

Seen again https://app.circleci.com/pipelines/github/mozilla/experimenter/41133/workflows/edf4e012-6cae-4bfe-9c05-bfd33f773b30/jobs/205826