Test failure with state session

[wwwillchen]

Test failure: https://github.com/google/mesop/actions/runs/10606188926/job/29396449678?pr=874

Looks like there's some kind of issue/bug with state sessions:

  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/mesop/mesop/server/state_session.py", line 93, in clear_stale_sessions
    for key, (timestamp, _) in self.cache.items():
RuntimeError: dictionary changed size during iteration

Stack trace:

°·····························°·················································
[WebServer] [WebServer] E0829 00:12:14.158636 139957666571840 _internal.py:97] Error on request:
Traceback (most recent call last):
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_werkzeug/site-packages/werkzeug/serving.py", line 363, in run_wsgi
    execute(self.server.app)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_werkzeug/site-packages/werkzeug/serving.py", line 324, in execute
    application_iter = app(environ, start_response)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714[57](https://github.com/google/mesop/actions/runs/10606188926/job/29396449678?pr=874#step:14:58)1e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_flask/site-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff[59](https://github.com/google/mesop/actions/runs/10606188926/job/29396449678?pr=874#step:14:60)ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_flask/site-packages/flask/app.py", line 1489, in wsgi_app
    ctx.pop(error)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_flask/site-packages/flask/ctx.py", line 410, in pop
    self.app.do_teardown_request(exc)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/my_deps_flask/site-packages/flask/app.py", line 1318, in do_teardown_request
    self.ensure_sync(func)(exc)
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/mesop/mesop/server/server.py", line 274, in teardown_clear_stale_state_sessions
    runtime().context().clear_stale_state_sessions()
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/mesop/mesop/runtime/context.py", line 246, in clear_stale_state_sessions
    state_session.clear_stale_sessions()
  File "/home/runner/.cache/bazel/_bazel_runner/283da4fe714571e9f73edeff59ade942/execroot/mesop/bazel-out/k8-fastbuild/bin/mesop/cli/cli.runfiles/mesop/mesop/server/state_session.py", line 93, in clear_stale_sessions
    for key, (timestamp, _) in self.cache.items():
RuntimeError: dictionary changed size during iteration
················T

  1) [chromium] › mesop/tests/e2e/web_components/custom_font_csp_test.ts:3:5 › web components - make sure custom font does not cause CSP issues 

    Test timeout of 75000ms exceeded.

    Error: locator.textContent: Test timeout of [75](https://github.com/google/mesop/actions/runs/10606188926/job/29396449678?pr=874#step:14:76)000ms exceeded.
    Call log:
      - waiting for getByText('Custom font: ')

       6 |   await page.goto('/web_component/custom_font_csp_repro/custom_font_app');
       7 |   // Make sure page has loaded.
    >  8 |   expect(await page.getByText('Custom font: ').textContent()).toEqual(
         |                                                ^
       9 |     'Custom font: Inter Tight',
      10 |   );
      11 |

        at /home/runner/work/mesop/mesop/mesop/tests/e2e/web_components/custom_font_csp_test.ts:8:48

[richard-to]

Hmm, I wonder if it's a concurrency issue. The loop in question does not modify anything. But if clear_stale_sessions is called by multiple requests around the same time, I imagine it could be possible for a race condition. I'll look into it more later.

[wwwillchen]

Yeah, that's what I was thinking to. If you do --repeat-each=10 for the whole test suite, you might be able to repro this issue.