collerek / ormar

python async orm with fastapi in mind and pydantic validation
https://collerek.github.io/ormar/
MIT License
1.68k stars 89 forks source link

`load_all()`/`select_related()` pydantic validation errors for ormar.JSON() field type #1391

Open dingxuanyao opened 3 months ago

dingxuanyao commented 3 months ago

Describe the bug When running load_all() or select_related() on model with foreign model with ormar.JSON() field, we see a pydantic validation error.

To Reproduce

import asyncio
import databases

import ormar
import sqlalchemy

DATABASE_URL = "sqlite:///db.sqlite"
base_ormar_config = ormar.OrmarConfig(
    database=databases.Database(DATABASE_URL),
    metadata=sqlalchemy.MetaData(),
    engine=sqlalchemy.create_engine(DATABASE_URL),
)

class Author(ormar.Model):
    ormar_config = base_ormar_config.copy(tablename="authors")
    id: int = ormar.Integer(primary_key=True)

class Book(ormar.Model):
    ormar_config = base_ormar_config.copy(tablename="books")
    id: int = ormar.Integer(primary_key=True)
    author: Author = ormar.ForeignKey(Author, name="author_id")
    my_data: dict = ormar.JSON(nullable=True)

async def main():
    base_ormar_config.metadata.drop_all(base_ormar_config.engine)
    base_ormar_config.metadata.create_all(base_ormar_config.engine)

    author = await Author.objects.create()
    book = await Book.objects.create(
        author=author,
        my_data={}
    )
    # this errors out
    authors =  await Author.objects.select_related(Author.books).all()
    # this also errors out
    authors = await Author.objects.select_all().all()

asyncio.run(main())
Error traceback

``` Traceback (most recent call last): File "/Users/tonyyao/workspace/ormar-bug/main.py", line 50, in asyncio.run(main()) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/Users/tonyyao/workspace/ormar-bug/main.py", line 47, in main authors = await Author.objects.select_all().all() File "/Users/tonyyao/.local/pipx/.cache/f0aff6fe1a77366/lib/python3.9/site-packages/ormar/queryset/queryset.py", line 1084, in all result_rows = await self._process_query_result_rows(rows) File "/Users/tonyyao/.local/pipx/.cache/f0aff6fe1a77366/lib/python3.9/site-packages/ormar/queryset/queryset.py", line 189, in _process_query_result_rows self.model.from_row( File "/Users/tonyyao/.local/pipx/.cache/f0aff6fe1a77366/lib/python3.9/site-packages/ormar/models/model_row.py", line 104, in from_row instance = cast("Model", cls(**item)) File "/Users/tonyyao/.local/pipx/.cache/f0aff6fe1a77366/lib/python3.9/site-packages/ormar/models/newbasemodel.py", line 138, in __init__ self.__pydantic_validator__.validate_python( pydantic_core._pydantic_core.ValidationError: 12 validation errors for Author books.int Input should be a valid integer [type=int_type, input_value=Book({'id': 1, 'author': ...040>]}), 'my_data': {}}), input_type=Book] For further information visit https://errors.pydantic.dev/2.5/v/int_type books.Book.my_data JSON input should be string, bytes or bytearray [type=json_type, input_value={}, input_type=dict] For further information visit https://errors.pydantic.dev/2.5/v/json_type books.PkOnlyBookrybjdp Input should be a valid dictionary or instance of PkOnlyBookrybjdp [type=model_type, input_value=Book({'id': 1, 'author': ...040>]}), 'my_data': {}}), input_type=Book] For further information visit https://errors.pydantic.dev/2.5/v/model_type books.`list[nullable[union[int,Book,...]]]`.0.int Input should be a valid integer [type=int_type, input_value=('id', 1), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/int_type books.`list[nullable[union[int,Book,...]]]`.0.Book Input should be a valid dictionary or object to extract fields from [type=model_attributes_type, input_value=('id', 1), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_attributes_type books.`list[nullable[union[int,Book,...]]]`.0.PkOnlyBookrybjdp Input should be a valid dictionary or instance of PkOnlyBookrybjdp [type=model_type, input_value=('id', 1), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_type books.`list[nullable[union[int,Book,...]]]`.1.int Input should be a valid integer [type=int_type, input_value=('author', Author({'id': ...Book at 0x102391040>]})), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/int_type books.`list[nullable[union[int,Book,...]]]`.1.Book Input should be a valid dictionary or object to extract fields from [type=model_attributes_type, input_value=('author', Author({'id': ...Book at 0x102391040>]})), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_attributes_type books.`list[nullable[union[int,Book,...]]]`.1.PkOnlyBookrybjdp Input should be a valid dictionary or instance of PkOnlyBookrybjdp [type=model_type, input_value=('author', Author({'id': ...Book at 0x102391040>]})), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_type books.`list[nullable[union[int,Book,...]]]`.2.int Input should be a valid integer [type=int_type, input_value=('my_data', {}), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/int_type books.`list[nullable[union[int,Book,...]]]`.2.Book Input should be a valid dictionary or object to extract fields from [type=model_attributes_type, input_value=('my_data', {}), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_attributes_type books.`list[nullable[union[int,Book,...]]]`.2.PkOnlyBookrybjdp Input should be a valid dictionary or instance of PkOnlyBookrybjdp [type=model_type, input_value=('my_data', {}), input_type=tuple] For further information visit https://errors.pydantic.dev/2.5/v/model_type ```

Expected behavior Unless I'm using these functions the wrong way, I expect either of these functions to not hit any pydantic errors.

Versions (please complete the following information):

# /// script
# dependencies = [
#   "databases[aiosqlite]==0.7.0",
#   "pydantic==2.5.3",
#   "ormar==0.20.1",
#   "sqlalchemy==1.4.52"
# ]
# ///

Additional context

Thanks in advance!

tsalpekar21 commented 2 months ago

Having this same issue. Glad you were able to figure out it was related to a JSON field!