Pydantic v2 - performance issues on import

criemen commented 8 months ago

Hi,

when testing the pydantic v2 prelease (0.11.0a0), we're seeing a big performance decrease on process startup time. My test file:

from githubkit import GitHub

went from ca. 1.6sec to 3.8sec on my M1 macbook. On production systems that have less powerful CPUs and slower I/Os, we're seeing this take 15-20sec now.

When interrupting the loading process with Ctrl+C, we see a stacktrace ala which hints at pydantic being the issue.

  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/__init__.py", line 2, in <module>
    from .github import GitHub as GitHub
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/github.py", line 16, in <module>
    from .core import GitHubCore
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/core.py", line 25, in <module>
    from .auth import BaseAuthStrategy, TokenAuthStrategy, UnauthAuthStrategy
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/auth/__init__.py", line 1, in <module>
    from .app import AppAuthStrategy as AppAuthStrategy
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/auth/app.py", line 10, in <module>
    from githubkit.rest import (
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/rest/__init__.py", line 13, in <module>
    from .models import *
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/githubkit/rest/models.py", line 17358, in <module>
    class ReposOwnerRepoCodespacesGetResponse200(GitHubRestModel):
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 184, in __new__
    complete_model_class(
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 481, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/main.py", line 576, in __get_pydantic_core_schema__
    return __handler(__source)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 452, in generate_schema
    schema = self._generate_schema(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 684, in _generate_schema
    schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 706, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 870, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 923, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1592, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1682, in new_handler
    schema = metadata_get_schema(source, get_inner_schema)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_std_types_schema.py", line 315, in __get_pydantic_core_schema__
    items_schema = handler.generate_schema(self.item_source_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 96, in generate_schema
    return self._generate_schema.generate_schema(__source_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 447, in generate_schema
    from_property = self._generate_schema_from_property(obj, obj)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 600, in _generate_schema_from_property
    schema = get_schema(
             ^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/main.py", line 576, in __get_pydantic_core_schema__
    return __handler(__source)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 684, in _generate_schema
    schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 706, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 870, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 923, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1592, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1571, in inner_handler
    from_property = self._generate_schema_from_property(obj, obj)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 600, in _generate_schema_from_property
    schema = get_schema(
             ^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/main.py", line 576, in __get_pydantic_core_schema__
    return __handler(__source)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 684, in _generate_schema
    schema = self._post_process_generated_schema(self._generate_schema_inner(obj))
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 706, in _generate_schema_inner
    return self._model_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 525, in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 870, in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 923, in _common_field_schema
    schema = self._apply_annotations(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1592, in _apply_annotations
    schema = get_inner_schema(source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in __call__
    schema = self._handler(__source_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1576, in inner_handler
    metadata_js_function = _extract_get_pydantic_json_schema(obj, schema)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/criemen/repos/codeql-halo/.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1924, in _extract_get_pydantic_json_schema
    js_modify_function = getattr(tp, '__get_pydantic_json_schema__', None)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.4/Frameworks/Python.framework/Versions/3.11/lib/python3.11/typing.py", line 1280, in __getattr__
    def __getattr__(self, attr):

Is there anything that can be done about this?

Upvote & Fund

We're using Polar.sh so you can upvote and help fund this issue.
We receive the funding once the issue is completed & confirmed by you.
Thank you in advance for helping prioritize & fund our backlog.

criemen commented 8 months ago

On top of that, RAM usage after loading githubkit is significantly higher with pydantic v2 as well.

For our gunicorn server (4 process), with 0.10.7 we see a baseline of <800MB, whereas with githubkit 0.11.0a0 we see 2.3GB.

yanyongyu commented 8 months ago

Due to the huge number of models, i decide to use lazy import to improve the performance. #11

The models will be loaded when the api first time imported.

To lazy import the models, the codegen should be changed to group the models for each api version and each api client (api scope such as repo, issues)

criemen commented 8 months ago

I investigated this a bit more: Some things that help (but model building is still slow!) are:

Upgrading to pydantic 2.4.2 - there are some potential performance improvements in there. It's not amazing, and you'll run into https://github.com/pydantic/pydantic/issues/7960 when upgrading
Switch from update_forward_refs() to model_rebuild() in models.py - the former is deprecated, and calls the latter with force=True, so the latter is doing less work
As you're calling model_rebuild on all models anyways, you can experiment with setting "defer_build": True for all models through the config settings in GithubRestModel

frankie567 commented 6 months ago

Hey @yanyongyu 👋 Since we benefit a lot from your work at polarsource/polar, I just pledged 100$ on this issue! Feel free to claim it over at https://polar.sh/ 😄

yanyongyu commented 6 months ago

Hello, i'm currently working on lazy importing and api versioning. you can see it on the https://github.com/yanyongyu/githubkit/tree/feat/codegen-refactor branch. The first step which i'm working on is to refactor the codegen to support api versioning and top-level lazy loading (by defer import).

yanyongyu commented 6 months ago

Hi @criemen @frankie567 , i have just published a release 0.11.0a2 with fully lazy loading feature. You can test it by installing the pre-release.

yanyongyu / githubkit

Pydantic v2 - performance issues on import #54

Upvote & Fund