Use of pydantic custom field validator returns HTTP 500 instead of HTTP 400

[peterschutt]

Discussed in https://github.com/orgs/litestar-org/discussions/2363

^{Originally posted by **trcw** September 26, 2023} I have a pydantic model with a custom validator that throws a ```ValueError``` as described in https://docs.pydantic.dev/latest/errors/errors/#custom-errors. The behavior is different in version 1 and 2 when I try to do an invalid post request. * In Starlite 1.51.14 the request returns ```HTTP 400``` with details of the validation failure. * In Litestar 2.1.0 the request returns ```HTTP 500``` with an internal server error instead. I was expecting a ```HTTP 400```. Is this a regression or do I need to do something different? Here is a minimal example for Litestar v2.1.0: ```python from litestar import Litestar, post from litestar.contrib.pydantic import PydanticDTO from litestar.testing import TestClient from pydantic import BaseModel, field_validator class User(BaseModel): user_id: int @field_validator('user_id') @classmethod def user_id_must_be_greater_than_zero(cls, user_id): if user_id < 1: raise ValueError('user id must be greater than 0') return user_id UserDTO = PydanticDTO[User] @post("/user", dto=UserDTO, sync_to_thread=False) def create_user(data: User) -> User: return data with TestClient(Litestar([create_user], debug=True)) as client: response = client.post("/user", json={"user_id": 0}) print(response.text) print(f"Status code: {response.status_code}") assert response.status_code == 400 ``` ``` Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 49, in decode_bytes return super().decode_bytes(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/base_dto.py", line 96, in decode_bytes return backend.populate_data_from_raw(value, self.asgi_connection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 297, in populate_data_from_raw return _transfer_data( ^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 557, in _transfer_data return _transfer_instance_data( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 621, in _transfer_instance_data return destination_type(**unstructured_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/pydantic/main.py", line 165, in __init__ __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) pydantic_core._pydantic_core.ValidationError: 1 validation error for User user_id Value error, user id must be greater than 0 [type=value_error, input_value=0, input_type=int] For further information visit https://errors.pydantic.dev/2.3/v/value_error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 79, in handle response = await self._get_response_for_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 131, in _get_response_for_request response = await self._call_handler_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 160, in _call_handler_function response_data, cleanup_group = await self._get_response_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 184, in _get_response_data kwargs["data"] = await kwargs["data"] ^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/_kwargs/extractors.py", line 427, in dto_extractor return data_dto(connection).decode_bytes(body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 51, in decode_bytes raise ValidationException(extra=ex.errors()) from ex litestar.exceptions.http_exceptions.ValidationException: 400: Bad Request Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 49, in decode_bytes return super().decode_bytes(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/base_dto.py", line 96, in decode_bytes return backend.populate_data_from_raw(value, self.asgi_connection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 297, in populate_data_from_raw return _transfer_data( ^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 557, in _transfer_data return _transfer_instance_data( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 621, in _transfer_instance_data return destination_type(**unstructured_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/pydantic/main.py", line 165, in __init__ __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) pydantic_core._pydantic_core.ValidationError: 1 validation error for User user_id Value error, user id must be greater than 0 [type=value_error, input_value=0, input_type=int] For further information visit https://errors.pydantic.dev/2.3/v/value_error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 79, in handle response = await self._get_response_for_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 131, in _get_response_for_request response = await self._call_handler_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 160, in _call_handler_function response_data, cleanup_group = await self._get_response_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 184, in _get_response_data kwargs["data"] = await kwargs["data"] ^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/_kwargs/extractors.py", line 427, in dto_extractor return data_dto(connection).decode_bytes(body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 51, in decode_bytes raise ValidationException(extra=ex.errors()) from ex litestar.exceptions.http_exceptions.ValidationException: 400: Bad Request ERROR - 2023-09-25 15:57:01,967 - litestar - config - exception raised on http connection to route /user Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 131, in _get_response_for_request response = await self._call_handler_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 160, in _call_handler_function response_data, cleanup_group = await self._get_response_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 184, in _get_response_data kwargs["data"] = await kwargs["data"] ^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/_kwargs/extractors.py", line 427, in dto_extractor return data_dto(connection).decode_bytes(body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 51, in decode_bytes raise ValidationException(extra=ex.errors()) from ex litestar.exceptions.http_exceptions.ValidationException: 400: Bad Request During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 141, in encode_json return msgspec.json.encode(value, enc_hook=serializer) if serializer else _msgspec_json_encoder.encode(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 88, in default_serializer raise TypeError(f"Unsupported type: {type(value)!r}") TypeError: Unsupported type: The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 205, in __call__ await self.handle_request_exception( File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 235, in handle_request_exception await response.to_asgi_response(app=None, request=request)(scope=scope, receive=receive, send=send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 444, in to_asgi_response body=self.render(self.content, media_type, get_serializer(type_encoders)), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 385, in render return encode_json(content, enc_hook) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 143, in encode_json raise SerializationException(str(msgspec_error)) from msgspec_error litestar.exceptions.base_exceptions.SerializationException: Unsupported type: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 49, in decode_bytes return super().decode_bytes(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/base_dto.py", line 96, in decode_bytes return backend.populate_data_from_raw(value, self.asgi_connection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 297, in populate_data_from_raw return _transfer_data( ^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 557, in _transfer_data return _transfer_instance_data( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 621, in _transfer_instance_data return destination_type(**unstructured_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/pydantic/main.py", line 165, in __init__ __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) pydantic_core._pydantic_core.ValidationError: 1 validation error for User user_id Value error, user id must be greater than 0 [type=value_error, input_value=0, input_type=int] For further information visit https://errors.pydantic.dev/2.3/v/value_error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 79, in handle response = await self._get_response_for_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 131, in _get_response_for_request response = await self._call_handler_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 160, in _call_handler_function response_data, cleanup_group = await self._get_response_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 184, in _get_response_data kwargs["data"] = await kwargs["data"] ^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/_kwargs/extractors.py", line 427, in dto_extractor return data_dto(connection).decode_bytes(body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 51, in decode_bytes raise ValidationException(extra=ex.errors()) from ex litestar.exceptions.http_exceptions.ValidationException: 400: Bad Request During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 141, in encode_json return msgspec.json.encode(value, enc_hook=serializer) if serializer else _msgspec_json_encoder.encode(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 88, in default_serializer raise TypeError(f"Unsupported type: {type(value)!r}") TypeError: Unsupported type: The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 205, in __call__ await self.handle_request_exception( File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 235, in handle_request_exception await response.to_asgi_response(app=None, request=request)(scope=scope, receive=receive, send=send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 444, in to_asgi_response body=self.render(self.content, media_type, get_serializer(type_encoders)), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 385, in render return encode_json(content, enc_hook) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 143, in encode_json raise SerializationException(str(msgspec_error)) from msgspec_error litestar.exceptions.base_exceptions.SerializationException: Unsupported type: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 49, in decode_bytes return super().decode_bytes(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/base_dto.py", line 96, in decode_bytes return backend.populate_data_from_raw(value, self.asgi_connection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 297, in populate_data_from_raw return _transfer_data( ^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 557, in _transfer_data return _transfer_instance_data( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/dto/_backend.py", line 621, in _transfer_instance_data return destination_type(**unstructured_data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/pydantic/main.py", line 165, in __init__ __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) pydantic_core._pydantic_core.ValidationError: 1 validation error for User user_id Value error, user id must be greater than 0 [type=value_error, input_value=0, input_type=int] For further information visit https://errors.pydantic.dev/2.3/v/value_error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 79, in handle response = await self._get_response_for_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 131, in _get_response_for_request response = await self._call_handler_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 160, in _call_handler_function response_data, cleanup_group = await self._get_response_data( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/routes/http.py", line 184, in _get_response_data kwargs["data"] = await kwargs["data"] ^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/_kwargs/extractors.py", line 427, in dto_extractor return data_dto(connection).decode_bytes(body) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_dto_factory.py", line 51, in decode_bytes raise ValidationException(extra=ex.errors()) from ex litestar.exceptions.http_exceptions.ValidationException: 400: Bad Request During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 141, in encode_json return msgspec.json.encode(value, enc_hook=serializer) if serializer else _msgspec_json_encoder.encode(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 88, in default_serializer raise TypeError(f"Unsupported type: {type(value)!r}") TypeError: Unsupported type: The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__ await self.app(scope, receive, send) File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 205, in __call__ await self.handle_request_exception( File "/home/vscode/.local/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 235, in handle_request_exception await response.to_asgi_response(app=None, request=request)(scope=scope, receive=receive, send=send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 444, in to_asgi_response body=self.render(self.content, media_type, get_serializer(type_encoders)), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/response/base.py", line 385, in render return encode_json(content, enc_hook) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/litestar/serialization/msgspec_hooks.py", line 143, in encode_json raise SerializationException(str(msgspec_error)) from msgspec_error litestar.exceptions.base_exceptions.SerializationException: Unsupported type: INFO - 2023-09-25 15:57:01,972 - httpx - _client - HTTP Request: POST http://testserver.local/user "HTTP/1.1 500 Internal Server Error" Status code: 500 Traceback (most recent call last): File "/workspaces/busshark.controller.trafficstore-api/main.py", line 28, in assert response.status_code == 400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError ```

---

Funding

• If you would like to see an issue prioritized, make a pledge towards it!
• We receive the pledge once the issue is completed & verified

[tuukkamustonen]

Reproduces on litestar==2.5.1 / pydantic==2.5.3.

[provinzkraut]

Turns out this isn't really a bug as such and has nothing to do with the field_validator in particular. It's just that Pydantic v2 return the original ValueError raised as part of its own exception when formatted as a dict. Since we in turn pass this to our ValidationException, and later on try to serialize it, we're hitting a serialization error because we can't actually serialize ValueError.

The reason why this only affects DTOs is because they take a very different error handling path; This also results in slightly different error responses when DTOs are used.

@peterschutt I think we should tackle this for v3 as it can only be done with breaking changes.

[github-actions[bot]]

This issue has been closed in #3286. The change will be included in the upcoming patch release.

[github-actions[bot]]

A fix for this issue has been released in v2.8.0