Pydantic.PrivateAttr `default` and `default_factory` are ignored by SQLModel

JLHasson commented 3 years ago

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the SQLModel documentation, with the integrated search.
[X] I already searched in Google "How to X in SQLModel" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to SQLModel but to Pydantic.
[X] I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code

from typing import Optional
from pydantic import BaseModel, PrivateAttr

from sqlmodel import Field, Session, SQLModel, create_engine, select

class Hero(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    _name: str = PrivateAttr(default=None)  # This field is not committed to the db
    secret_name: str

class HeroPydantic(BaseModel):
    _name: str = PrivateAttr(default=None)
    secret_name: str

sqlite_file_name = "database.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"

engine = create_engine(sqlite_url, echo=True)

if __name__ == "__main__":
    SQLModel.metadata.create_all(engine)

    hero_1 = Hero(secret_name="Dive Wilson")
    print("Hero:", hero_1)
    print(hero_1._name)
    print(hero_1._name is None, type(hero_1._name))

    hero_2 = HeroPydantic(secret_name="Lance")
    print(hero_2)
    print(hero_2._name)
    print(hero_2._name is None, type(hero_2._name))

    with Session(engine) as session:
        session.add(hero_1)
        session.commit()
        statement = select(Hero)
        results = session.exec(statement)
        for hero in results:
            print(hero)
            print(hero_2._name)
            print(hero_2._name is None, type(hero_2._name))

Description

As far as I can tell SQLModel is ignoring the default and default_factory parameters of pydantic.PrivateAttr. The example I've given above reproduces on my system. The output can be seen here:

Hero: id=None secret_name='Dive Wilson'

False <class 'pydantic.fields.ModelPrivateAttr'>
secret_name='Lance'
None
True <class 'NoneType'>
2021-10-28 12:17:30,129 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2021-10-28 12:17:30,131 INFO sqlalchemy.engine.Engine INSERT INTO hero (secret_name) VALUES (?)
2021-10-28 12:17:30,131 INFO sqlalchemy.engine.Engine [generated in 0.00015s] ('Dive Wilson',)
2021-10-28 12:17:30,131 INFO sqlalchemy.engine.Engine COMMIT
2021-10-28 12:17:30,143 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2021-10-28 12:17:30,144 INFO sqlalchemy.engine.Engine SELECT hero.id, hero.secret_name
FROM hero
2021-10-28 12:17:30,144 INFO sqlalchemy.engine.Engine [no key 0.00010s] ()
secret_name='Dive Wilson' id=1

False <class 'pydantic.fields.ModelPrivateAttr'>
2021-10-28 12:17:30,144 INFO sqlalchemy.engine.Engine ROLLBACK

As you can see the field is not set to None, and instead is an empty instance of pydantic.fields.ModelPrivateAttr.

Operating System

macOS

Operating System Details

No response

SQLModel Version

0.0.4

Python Version

3.9.5

Additional Context

No response

ChielWH commented 2 years ago

Tried some other options for achieving this, with a pydantic model as the PrivateAttr. The solution as shown in https://github.com/tiangolo/sqlmodel/issues/147 doens't work either. Interestingly, running this in a notebook yields the expected result for B3.

Environment: Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-87-generic x86_64) Python 3.8.12 pydantic==1.9.0 sqlmodel==0.0.6 ipykernel==6.9.0

from typing import Optional
from pydantic import BaseModel, PrivateAttr
from sqlmodel import Field, SQLModel

class A(BaseModel):
    some_attr: int = 2

class B1(SQLModel, table=True):
    _a: A = PrivateAttr(default_factory=A)
    id: Optional[int] = Field(default=None, primary_key=True)

class B2(SQLModel, table=True):
    _a: A = PrivateAttr(default=A())
    id: Optional[int] = Field(default=None, primary_key=True)

class B3(SQLModel, table=True):
    _a: A = PrivateAttr()
    id: Optional[int] = Field(default=None, primary_key=True)

    # from https://pydantic-docs.helpmanual.io/usage/models/#private-model-attributes
    def __init__(self, **data):
        super().__init__(**data)
        self._a = A()

class B4(SQLModel, table=True):
    _a: A = PrivateAttr()
    id: Optional[int] = Field(default=None, primary_key=True)

    @property
    def a(self):
        return self._a

for B in [B1, B2, B3, B4]:
    b = B()
    try:
        print(b._a.some_attr)
    except Exception as e:
        print(e)

Result in plain python:

'ModelPrivateAttr' object has no attribute 'some_attr'
'ModelPrivateAttr' object has no attribute 'some_attr'
'ModelPrivateAttr' object has no attribute 'some_attr'
'ModelPrivateAttr' object has no attribute 'some_attr'

Result in a notebook:

'ModelPrivateAttr' object has no attribute 'some_attr'
'ModelPrivateAttr' object has no attribute 'some_attr'
2
'ModelPrivateAttr' object has no attribute 'some_attr'

Corfucinas commented 2 years ago

Any solution for this? I just ran into the same problem

alexisgaziello commented 2 years ago

It appears that we are missing private attribute initialization. I have raised a PR to fix this.

Corfucinas commented 2 years ago

Great! I ended up adding 8 columns to my SQL table because of this bug.., finally able to clean up after this merge

swan-alexchen commented 4 months ago

same issue here

robinelvin commented 3 months ago

I have a similar related issue. I have a PrivateAttr field on my model so it is not persisted with a public getter for clarity. If I construct the model myself the getter works but if I retrieve from the database with select then accessing the getter raises:

  File "/Volumes/Crucial X8/Projects/Portal/.venv/lib/python3.12/site-packages/pydantic/main.py", line 807, in __getattr__
    return self.__pydantic_private__[item]  # type: ignore
           │    │                    └ '_anonymous'
           │    └ <member '__pydantic_private__' of 'BaseModel' objects>
           └ User(id=1, username='testuser')

TypeError: 'NoneType' object is not subscriptable

The model

class User(SQLModel, table=True):
    __tablename__ = "user"

    id: int | None = Field(default=None, primary_key=True, gt=0)
    username: str = Field(index=True, max_length=32)
    email: str
    password: str = Field(repr=False)
    superuser: bool = Field(default=False)

    _anonymous: bool = PrivateAttr(default=False)

    @property
    def is_anonymous(self) -> bool:
        return self._anonymous

If I change my repository method to the following it works fine. Obviously not a viable solution going forward but serves to prove the point.

    def find_by_username(self, username: str) -> User | None:
        stmt = select(User).where(User.username == username)
        result = self.session.exec(stmt)
        r = result.one_or_none()
        u = User(**r.model_dump())
        return u

gravesee commented 3 weeks ago

I have a similar related issue. I have a PrivateAttr field on my model so it is not persisted with a public getter for clarity. If I construct the model myself the getter works but if I retrieve from the database with select then accessing the getter raises:

  File "/Volumes/Crucial X8/Projects/Portal/.venv/lib/python3.12/site-packages/pydantic/main.py", line 807, in __getattr__
    return self.__pydantic_private__[item]  # type: ignore
           │    │                    └ '_anonymous'
           │    └ <member '__pydantic_private__' of 'BaseModel' objects>
           └ User(id=1, username='testuser')

TypeError: 'NoneType' object is not subscriptable

The model

class User(SQLModel, table=True):
    __tablename__ = "user"

    id: int | None = Field(default=None, primary_key=True, gt=0)
    username: str = Field(index=True, max_length=32)
    email: str
    password: str = Field(repr=False)
    superuser: bool = Field(default=False)

    _anonymous: bool = PrivateAttr(default=False)

    @property
    def is_anonymous(self) -> bool:
        return self._anonymous

If I change my repository method to the following it works fine. Obviously not a viable solution going forward but serves to prove the point.

    def find_by_username(self, username: str) -> User | None:
        stmt = select(User).where(User.username == username)
        result = self.session.exec(stmt)
        r = result.one_or_none()
        u = User(**r.model_dump())
        return u

Furthermore, if you update the object and add it to the session sqlmodel considers it a new instance entirely and you will see a unique constraint error for the duplicate ID.

fastapi / sqlmodel