search

matrix-org / synapse

Synapse: Matrix homeserver written in Python/Twisted.
https://matrix-org.github.io/synapse
Apache License 2.0
11.84k stars 2.12k forks source link

Investigate if we can improve Pydantic's error messages for user consumption #13337

Open DMRobertson opened 2 years ago

DMRobertson commented 2 years ago

If anyone wants to see the kind of error messages Pydantic produces, try running this test case on my other pydantic experiment branch.

I think the name of the model isn't user-relevant. The "error context" (stuff in brackets) also loks like noise to an end-user. But we should be able to inspect the ValidationError and format it as we please.

/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-7yaa6cKe-py3.10/bin/python /home/dmr/.local/share/JetBrains/Toolbox/apps/PyCharm-C/ch-0/221.5787.24/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py --path /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py
Testing started at 19:38 ...
Launching unittests with arguments python -m unittest /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py in /home/dmr/workspace/synapse-2/tests/config

Ran 13 tests in 0.033s

OK

Process finished with exit code 0

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements -> 0
  value is not a valid dict (type=type_error.dict)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)

2 validation errors for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> answer
  extra fields not permitted (type=value_error.extra)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  expected string or bytes-like object (type=type_error)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI '' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'notaurl' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'https://example.com' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'mxc://mxc://mxc://' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_id
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at least 1 characters (type=value_error.any_str.min_length; limit_value=1)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
idp_id
  string does not match regex "^[A-Za-z0-9._~-]+$" (type=value_error.str.regex; pattern=^[A-Za-z0-9._~-]+$)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  field required (type=value_error.missing)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)
idp_name
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

Originally posted by @DMRobertson in https://github.com/matrix-org/synapse/issues/13188#issuecomment-1175382843

DMRobertson commented 2 years ago

Ahh, there is a model config option error_msg_templates which might be useful here. And we could set this globally by defining our own BaseModel, see https://pydantic-docs.helpmanual.io/usage/model_config/#change-behaviour-globally

Edit: I ended up creating RequestBodyModel

DMRobertson commented 2 years ago

Rich notes:

I don't love the multiline error messages, particularly in the logs (where it is useful to be able to grep ERROR and hope to see most of the useful text). It might be better just to complain about the first error rather than trying to list all of them (which is what the jsonschema-based validation for config settings does).

We can iterate on this in future though.