taskiq-python / taskiq

Distributed task queue with full async support
MIT License
689 stars 44 forks source link

Reuse TypeAdapter instance to improve parsing params performance #281

Closed unights closed 5 months ago

unights commented 5 months ago

According to the pydantic v2 document, the TypeAdapter should construct only once and reuse it.

Here is a simple test.

from datetime import date, datetime
from timeit import timeit

from pydantic import TypeAdapter

def parse_obj(annot, value):
    ta = TypeAdapter(annot)

    result1 = timeit(lambda: ta.validate_python(value), number=10000)
    result2 = timeit(lambda: TypeAdapter(annot).validate_python(value), number=10000)

    print(f'{annot.__name__:<8} {result2 / result1:.2f}')

if __name__ == '__main__':
    for t, v in (
        (int, "0"),
        (float, '0.5'),
        (str, b"123"),
        (bool, 'true'),
        (date, '2024-01-01'),
        (datetime, '2024-01-01T00:00:00'),
    ):
        parse_obj(t, v)

output is

int      225.59
float    140.84
str      133.88
bool     228.47
date     357.20
datetime 319.76

Now the params parser in taskiq is

# taskiq/compat.py
def parse_obj_as(annot: T, obj: Any) -> T:
    return pydantic.TypeAdapter(annot).validate_python(obj)

And small changes may improve its performance

from functools import lru_cache

@lru_cache
def get_adapter(annot: T) -> pydantic.TypeAdapter:
    return pydantic.TypeAdapter(annot)

def parse_obj_as(annot: T, obj: Any) -> T:
    return get_adapter(annot).validate_python(obj)
vvanglro commented 5 months ago

Great! PR welcome.