taskiq-python / taskiq

Distributed task queue with full async support
MIT License
806 stars 50 forks source link

TaskProgress PydanticSchemaGenerationError TaskProgress schema arbitrary types bug #336

Open joeygrable94 opened 3 months ago

joeygrable94 commented 3 months ago

I am getting the following error since upgrading to 11.5

The Pydantic package does not like the TaskProgress type in the progress tracker. The types were discussed in this PR previously.

When I go into the taskqi.depends.task_progress and add the config arbitrary_types_allow=True then this issue goes away. I am hoping someone can explain why this might be happening in my code!

pydantic = {version = "2.7.1", extras = ["email"]}

ImportError while loading conftest '/path/to/app/tests/conftest.py'.
tests/conftest.py:18: in <module>
    from app.main import create_app
app/main.py:103: in <module>
    app: FastAPI = create_app()
app/main.py:99: in create_app
    configure_routers(app)
app/main.py:56: in configure_routers
    from app.api.v1 import router_v1
app/api/v1/__init__.py:3: in <module>
    from app.api.v1.endpoints import (
app/api/v1/endpoints/data_feed.py:38: in <module>
    from app.tasks import task_create_client_data_bucket
app/tasks/__init__.py:1: in <module>
    from .core_tasks import (
app/tasks/core_tasks.py:2: in <module>
    from app.worker import task_broker
app/worker.py:1: in <module>
    from taskiq import AsyncBroker, AsyncResultBackend
.venv/lib/python3.11/site-packages/taskiq/__init__.py:12: in <module>
    from taskiq.brokers.inmemory_broker import InMemoryBroker
.venv/lib/python3.11/site-packages/taskiq/brokers/inmemory_broker.py:8: in <module>
    from taskiq.depends.progress_tracker import TaskProgress
.venv/lib/python3.11/site-packages/taskiq/depends/progress_tracker.py:28: in <module>
    class TaskProgress(GenericModel, Generic[_ProgressType]):
.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:202: in __new__
    complete_model_class(
.venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py:539: in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
.venv/lib/python3.11/site-packages/pydantic/main.py:626: in __get_pydantic_core_schema__
    return handler(source)
.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py:82: in __call__
    schema = self._handler(source_type)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:502: in generate_schema
    schema = self._generate_schema_inner(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:753: in _generate_schema_inner
    return self._model_schema(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:580: in _model_schema
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:580: in <dictcomp>
    {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:916: in _generate_md_field_schema
    common_field = self._common_field_schema(name, field_info, decorators)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1081: in _common_field_schema
    schema = self._apply_annotations(
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1820: in _apply_annotations
    schema = get_inner_schema(source_type)
.venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py:82: in __call__
    schema = self._handler(source_type)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1801: in inner_handler
    schema = self._generate_schema_inner(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:758: in _generate_schema_inner
    return self.match_type(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:840: in match_type
    return self._match_generic_type(obj, origin)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:864: in _match_generic_type
    return self._union_schema(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1152: in _union_schema
    choices.append(self.generate_schema(arg))
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:502: in generate_schema
    schema = self._generate_schema_inner(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:758: in _generate_schema_inner
    return self.match_type(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:817: in match_type
    return self._unsubstituted_typevar_schema(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:1667: in _unsubstituted_typevar_schema
    return self.generate_schema(default)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:502: in generate_schema
    schema = self._generate_schema_inner(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:758: in _generate_schema_inner
    return self.match_type(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:844: in match_type
    return self._unknown_type_schema(obj)
.venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py:405: in _unknown_type_schema
    raise PydanticSchemaGenerationError(
E   pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for typing_extensions.NoDefault. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
E
E   If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
E
E   For further information visit https://errors.pydantic.dev/2.7/u/schema-for-unknown-type
joeygrable94 commented 3 months ago

I created a PR with the class Config: arbitrary_types_allowed = True to pass Pydantic's type check on schema generation: https://github.com/taskiq-python/taskiq/pull/337