Python: Pydantic-core package doesnt consumes entropy

[morgan9e]

While running workerd with Python pydantic package, initilazing error RuntimeError: 1 unexpected leftover getentropy calls occurs. Seems fixing it into 0 getentropy call fixes problem, pydantic-core package doesnt consumes entropy?

Version is Pydantic-core 2.16.2

<module 'tempfile' from '/lib/python312.zip/tempfile.py'>
<module 'random' from '/lib/python312.zip/random.py'>
Loading fastapi, starlette, anyio, sniffio, openssl, pydantic, typing-extensions, pydantic-core, annotated-types
Loaded fastapi, starlette, anyio, sniffio, openssl, pydantic, typing-extensions, pydantic-core, annotated-types
<module 'pydantic_core' from '/session/lib/python3.12/site-packages/pydantic_core/__init__.py'>
Error in makeHandler
PythonError: Traceback (most recent call last):
  File "/lib/python312.zip/_pyodide/_base.py", line 629, in pyimport_impl
    res = __import__(stem, fromlist=fromlist)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/session/metadata/worker.py", line 1, in <module>
    from asgi import env
  File "/lib/python3.12/site-packages/asgi.py", line 4, in <module>
    from fastapi import Request, Depends
  File "/session/lib/python3.12/site-packages/fastapi/__init__.py", line 7, in <module>
    from .applications import FastAPI as FastAPI
  File "/session/lib/python3.12/site-packages/fastapi/applications.py", line 16, in <module>
    from fastapi import routing
  File "/session/lib/python3.12/site-packages/fastapi/routing.py", line 22, in <module>
    from fastapi import params
  File "/session/lib/python3.12/site-packages/fastapi/params.py", line 5, in <module>
    from fastapi.openapi.models import Example
  File "/session/lib/python3.12/site-packages/fastapi/openapi/models.py", line 4, in <module>
    from fastapi._compat import (
  File "/session/lib/python3.12/site-packages/fastapi/_compat.py", line 20, in <module>
    from fastapi.exceptions import RequestErrorModel
  File "/session/lib/python3.12/site-packages/fastapi/exceptions.py", line 3, in <module>
    from pydantic import BaseModel, create_model
  File "<frozen importlib._bootstrap>", line 1412, in _handle_fromlist
  File "/session/lib/python3.12/site-packages/pydantic/__init__.py", line 383, in __getattr__
    module = import_module(module_name, package=package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python312.zip/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/session/lib/python3.12/site-packages/pydantic/main.py", line 12, in <module>
    import pydantic_core
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/lib/python3.12/site-packages/_cloudflare/import_patch_manager.py", line 28, in exec_module
    with self.import_context(module):
  File "/lib/python312.zip/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/lib/python3.12/site-packages/_cloudflare/entropy_import_context.py", line 123, in pydantic_core_context
    with allow_bad_entropy_calls(1):
  File "/lib/python312.zip/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/lib/python3.12/site-packages/_cloudflare/entropy_import_context.py", line 48, in allow_bad_entropy_calls
    raise RuntimeError(
RuntimeError: 1 unexpected leftover getentropy calls 

    at new_error (pyodide-internal:generated/pyodide.asm:20:9998)
    at wasm://wasm/0265dd42:wasm-function[300]:0x16c818
    at wasm://wasm/0265dd42:wasm-function[301]:0x16c972
    at Module._pythonexc2js (pyodide-internal:generated/pyodide.asm:20:607956)
    at Module.callPyObjectKwargs (pyodide-internal:generated/pyodide.asm:20:64098)
    at Module.callPyObject (pyodide-internal:generated/pyodide.asm:20:65055)
    at Function.apply (pyodide-internal:generated/pyodide.asm:20:79132)
    at Object.apply (pyodide-internal:generated/pyodide.asm:20:77371)
    at Object.pyimport (pyodide-internal:generated/pyodide.asm:20:102862)
    at pyimportMainModule (pyodide:python-entrypoint-helper:31:18)
    at pyodide:python-entrypoint-helper:117:11
    at enterJaegerSpan (pyodide-internal:jaeger:9:12)
    at pyodide:python-entrypoint-helper:116:16
    at async preparePython (pyodide:python-entrypoint-helper:129:22)
    at async pyodide:python-entrypoint-helper:139:21
    at async Object.fetch (pyodide:python-entrypoint-helper:137:26)

Or am I using wrong version of Python or Pyodide?

[hoodmane]

This is really difficult to get right, but there are two places where it uses an entropy call:

1. the first time pyo3 makes a std::collections::HashMap, it initializes std::collections::hash_map::RandomState
2. pydantic-core itself uses ahash and the first time an ahash::AHashMap is constructed, ahash::RandomState makes an entropy call.

I wonder why we get different results when you run it than in our test suite.

[morgan9e]

@hoodmane: This is really difficult to get right, but there are two places where it uses an entropy call:

1. the first time pyo3 makes a std::collections::HashMap, it initializes [std::collections::hash_map::RandomState](https://doc.rust-lang.org/std/collections/hash_map/struct.RandomState.html)

2. pydantic-core itself uses [ahash](https://github.com/tkaitchuck/aHash/blob/master/src/hash_map.rs) and the first time an `ahash::AHashMap` is constructed, `ahash::RandomState` makes an entropy call.

I wonder why we get different results when you run it than in our test suite.

But even when I download latest build from runs/9421919278/job/25957049626, workerd and run ../workerd serve ./samples/pyodide-fastapi/config.capnp --verbose, I get

  File "/lib/python3.12/site-packages/_cloudflare/import_patch_manager.py", line 27, in exec_module
    with self.import_context(module):
  File "/lib/python312.zip/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/lib/python3.12/site-packages/_cloudflare/entropy_import_context.py", line 123, in pydantic_core_context
    with allow_bad_entropy_calls(1):
  File "/lib/python312.zip/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/lib/python3.12/site-packages/_cloudflare/entropy_import_context.py", line 48, in allow_bad_entropy_calls
    raise RuntimeError(
RuntimeError: 1 unexpected leftover getentropy calls 

So either its my local environment problem, or Pydantic-core has modified with version change?

I've tested Debian 12, Ubuntu 22.04, and Fedora 39.

[hoodmane]

Well we are testing this in our downstream internal test suite with this version of pydantic. I'm not sure what the reason for the discrepancy is I'll have to look into it. I added this logic because otherwise the test suite didn't pass. We should try to move more tests into the public repo so we have something to talk about.

[morgan9e]

Thanks, Then I'll try to look at pydantic-core for now if anything happened there.