`__pydantic_extra__` override triggers error in `pyright`

[awgymer]

Initial Checks

• [X] I confirm that I'm using Pydantic V2

Description

Overriding __pyright_extra__ causes a pyright typing error because the type is not the same as the base declaration.

Example Code

from pydantic import BaseModel, ConfigDict

ExtraDict = dict[str, int | str]

class ModelExtra(BaseModel):
    model_config = ConfigDict(extra="allow")
    __pydantic_extra__: ExtraDict = Field(init=False)

## Gives pyright error
#error: "__pydantic_extra__" overrides symbol of same name in class "BaseModel"
#    Variable is mutable so its type is invariant
#      Override type "ExtraDict" is not the same as base type "dict[str, Any] | None" (reportIncompatibleVariableOverride)

## BUT THIS ALSO BREAKS

class ModelExtra(BaseModel):
    model_config = ConfigDict(extra="allow")
    __pydantic_extra__: ExtraDict | None = Field(init=False)

## ERROR
# pydantic.errors.PydanticSchemaGenerationError: The type annotation for `__pydantic_extra__` must be `Dict[str, ...]`

Python, Pydantic & OS Version

pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
               python version: 3.10.4 (main, Oct 26 2022, 13:07:52) [Clang 13.0.0 (clang-1300.0.29.30)]
                     platform: macOS-15.0.1-arm64-arm-64bit
             related packages: mypy-1.6.1 pyright-1.1.385 typing_extensions-4.8.0
                       commit: unknown

[Viicos]

I think the API to specify validation for extra fields is a bit flawed. While specifying it as an annotation is beneficial as you can then easily type check attribute accesses to __pydantic_extra__, there's a couple issues arising from this:

• type checkers treat __pydantic_extra__ as a field, so using = Field(init=False) is necessary as an ugly workaround
• if you set extra to 'allow' without redefining __pydantic_extra__ (because you don't want any specific validation to be applied), __pydantic_extra__ is still annotated as dict[str, Any] | None, although at runtime it is guaranteed to not be None
• you get the the pyright error because it is theorically unsafe to narrow the annotation, as __pydantic_extra__ is considered mutable (see an explanation with a similar example here). In practice, I don't expect anyone to mutate the __pydantic_extra__ dictionary directly. In fact, we provide a model_extra property to access the extra values. It is also flawed because the type of this property does not reflect the possibly overridden __pydantic_extra__ annotation.

I can't come up with a good API immediately, however one thing that is certain is that we will have to break the existing API if we ever want to make changes here. As such, marking this issue as https://github.com/pydantic/pydantic/labels/v3.

In the meanwhile, you can add a pyright: ignore[reportIncompatibleVariableOverride] comment or disable the error code at the project level (enabling this error code is debatable, see https://github.com/microsoft/pyright/issues/5933).