Synapse does not take advantage of Pydantic 2.0

[c7hm4r]

Description

After upgrading the PIP packages, Synapse failed to launch.

Steps to reproduce

• install matrix-synapse using the "virtualenv" approach with the most recent dependency versions
• start Synapse

Homeserver

another homeserver

Synapse Version

1.87.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Single process

Platform

Raspberry Pi OS (Debian)

Configuration

No response

Relevant log output

/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:                                                            
* 'allow_mutation' has been removed                                                
  warnings.warn(message, UserWarning)                                              
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main             
    return _run_code(code, main_globals, None,                                     
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                        
    exec(code, run_globals)                                                        
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py", line 37, in <module>                                                                                         
    from synapse.app import _base                                                  
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/_base.py", line 68, in <module>                                                                                              
    from synapse.handlers.auth import load_legacy_password_auth_providers          
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/auth.py", line 56, in <module>                                                                                          
    from synapse.api.ratelimiting import Ratelimiter                              
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/api/ratelimiting.py", line 21, in <module>                                                                                       
    from synapse.storage.databases.main import DataStore                           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/__init__.py", line 34, in <module>                                                                                       
    from synapse.storage.databases import Databases                                
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 20, in <module>                                                                             
    from synapse.storage.databases.main.events import PersistEventsStore           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 27, in <module>                                                                        
    from synapse.storage.databases.main.stats import UserSortOrder                 
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 42, in <module>                                                                           
    from synapse.storage.databases.main.events_worker import InvalidEventError     
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_worker.py", line 62, in <module>
    from synapse.replication.tcp.streams import BackfillStream, UnPartialStatedEventStream
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/__init__.py", line 27, in <module>
    from synapse.replication.tcp.streams._base import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/_base.py", line 32, in <module>
    from synapse.replication.http.streams import ReplicationGetStreamUpdates
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/__init__.py", line 18, in <module>
    from synapse.replication.http import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/account_data.py", line 21, in <module>
    from synapse.replication.http._base import ReplicationEndpoint
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/_base.py", line 31, in <module>
    from synapse.http.servlet import parse_json_object_from_request
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/servlet.py", line 32, in <module>
    from pydantic import BaseModel, MissingError, PydanticValueError, ValidationError
ImportError: cannot import name 'MissingError' from 'pydantic' (/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/__init__.py)
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Anything else that would be useful to know?

pydantic 2.0 was released yesterday

[c7hm4r]

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

[DMRobertson]

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Suggest we specify this in pyproject.toml, or change

https://github.com/matrix-org/synapse/blob/07fd6d82d7890b10a02727c9892f01c684f207ef/pyproject.toml#L209-L210

to ^=1.7.4.

Not sure why I didn't do this when starting to use Pydantic. I think because generally speaking it makes packagers' lives easier if we don't impose upper bounds?

[PaarthShah]

Just encountered this, and in other projects as well on Friday morning. I agree with pinning to <2.0.0 as a workaround; hoping to make a PR for this shortly.

[PaarthShah]

Just created: https://github.com/matrix-org/synapse/pull/15862

[H-Shay]

Closing this as it looks like it is resolved by https://github.com/matrix-org/synapse/pull/15862

[dvzrv]

Hi! We're currently blocked by matrix-synapse in our pydantic 2 rebuild for Arch Linux. Please reopen this ticket so that discussion around how/when/if to switch to pydantic2 can happen.

The change in #15862 is merely a workaround and not the solution to this ticket (which would be "be compatible with pydantic >= 2").

Thanks! :)

[clokep]

Re-opening per the above -- pinning to <= 2.0 isn't supporting Pydantic 2.0.

[reivilibre]

Pydantic V2 includes V1 built-in to help migration.

We could use the following all over the place as a transitional measure.

try:
    from pydantic import v1 as pydantic
except ImportError:
    import pydantic

Given that V1 and V2 use the same Python package name (⇒ you can't have them both installed at the same time), we sadly can expect some trouble with distro packages that would make switching to using V2 proper difficult for some time.

[dvzrv]

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

I can earliest look at my open PR again after 2023-08-21. If someone manages to fix this earlier and create a new release, that would be much appreciated, as synapse is currently the only package blocking us from moving to pydantic v2 on Arch Linux: https://archlinux.org/todo/python-pydantic-2/

[gotmax23]

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

The pydantic.v1 module is simply a copy of the old pydantic v1 code, so there should not be breakages there.

[gotmax23]

I opened https://github.com/matrix-org/synapse/pull/16332 to add support for pydantic v2 while maintaining support for v1.

[clokep]

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

I re-titled this since Synapse is now compatible, but not taking advantage of pydantic v2.