Safe handler dictionary iteration

colebaileygit commented 3 months ago

I got this stack trace while running tests. I imagine it is a very rare race condition:

Can we make the handler data object safe for multi-threading?

tests/e2e/inprocess/grpc/test_inprocess_grpc_reconnect.py::test_provider_unavailable
  /Users/c.bailey/Library/Application Support/hatch/env/virtual/openfeature-provider-flagd/2sJ2q5Uj/openfeature-provider-flagd/lib/python3.9/site-packages/_pytest/threadexception.py:77: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-3

  Traceback (most recent call last):
    File "/Users/c.bailey/.pyenv/versions/3.9.13/lib/python3.9/threading.py", line 980, in _bootstrap_inner
      self.run()
    File "/Users/c.bailey/.pyenv/versions/3.9.13/lib/python3.9/threading.py", line 917, in run
      self._target(*self._args, **self._kwargs)
    File "/Users/c.bailey/Source/python-sdk-contrib/providers/openfeature-provider-flagd/src/openfeature/contrib/provider/flagd/resolvers/process/connector/grpc_watcher.py", line 95, in sync_flags
      self.provider.emit_provider_error(
    File "/Users/c.bailey/Library/Application Support/hatch/env/virtual/openfeature-provider-flagd/2sJ2q5Uj/openfeature-provider-flagd/lib/python3.9/site-packages/openfeature/provider/provider.py", line 81, in emit_provider_error
      self.emit(ProviderEvent.PROVIDER_ERROR, details)
    File "/Users/c.bailey/Library/Application Support/hatch/env/virtual/openfeature-provider-flagd/2sJ2q5Uj/openfeature-provider-flagd/lib/python3.9/site-packages/openfeature/provider/provider.py", line 87, in emit
      run_handlers_for_provider(self, event, details)
    File "/Users/c.bailey/Library/Application Support/hatch/env/virtual/openfeature-provider-flagd/2sJ2q5Uj/openfeature-provider-flagd/lib/python3.9/site-packages/openfeature/_event_support.py", line 75, in run_handlers_for_provider
      for client in _client_handlers:
  RuntimeError: dictionary changed size during iteration

    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

beeme1mr commented 3 months ago

Relates to https://github.com/open-feature/python-sdk/issues/96.

@colebaileygit would this be something you would be willing to help with? FYI @federicobond @gruebel @toddbaert

federicobond commented 3 months ago

Thanks for reporting @colebaileygit, it's nice to have an existence proof for the thread-safety issues we knew were there.

federicobond commented 3 months ago

I wrote a test that can reproduce the issue 100% of the time on my computer and implemented some locks to prevent concurrent access to the event handler dictionaries. See https://github.com/open-feature/python-sdk/pull/329

federicobond commented 3 months ago

There is still quite a lot of thread-safety work to be done with other methods, particularly in the openfeature.api module.

federicobond commented 3 months ago

I think we can mark this particular issue as resolved.

open-feature / python-sdk

Safe handler dictionary iteration #326