Bug: Nested multipart data not being parsed

[LonelyVikingMichael]

Description

Preface: I am trying to replicate functionality that was working in Starlite 1.51.x

It was previously possible to define nested models including UploadFile fields for consumption in a single request. Litestar raises a validation error on request.

URL to code causing the issue

No response

MCVE

from io import BytesIO
from json import dumps

from litestar import Litestar, post
from litestar.datastructures import UploadFile
from litestar.enums import RequestEncodingType
from litestar.params import Body
from litestar.testing import TestClient
from pydantic.v1 import BaseModel

class LineItem(BaseModel):
    cost: int
    description: str

class Order(BaseModel):
    file: UploadFile
    code: str
    line_items: list[LineItem]

    class Config:
        arbitrary_types_allowed = True

@post()
async def test_handler(data: Order = Body(media_type=RequestEncodingType.MULTI_PART)) -> dict:
    return data

app = Litestar(route_handlers=[test_handler], debug=True)

with TestClient(app=app) as client:
    response = client.post(
        "/",
        data={"code": "123", "line_items": dumps([{"cost": 5, "description": "Not free food"}])},
        files=[("file", ("name.jpg", BytesIO(b"todayisgood")))],
    )
    assert response.is_success

Steps to reproduce

Execute the MCVE

Screenshots

No response

Logs

ERROR - 2023-11-08 10:15:48,992 - litestar - config - exception raised on http connection to route /

Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_init_plugin.py", line 40, in _dec_pydantic_v1
    return model_type.parse_obj(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 526, in parse_obj
    return cls(**obj)
           ^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Order
line_items
  value is not a valid list (type=type_error.list)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 190, in parse_values_from_connection_kwargs
    return convert(kwargs, cls, strict=False, dec_hook=deserializer, str_keys=True).to_dict()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 91, in _deserializer
    return decoder(target_type, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_init_plugin.py", line 42, in _dec_pydantic_v1
    raise ExtendedMsgSpecValidationError(errors=cast("list[dict[str, Any]]", e.errors())) from e
litestar._signature.types.ExtendedMsgSpecValidationError: [{'loc': ('line_items',), 'msg': 'value is not a valid list', 'type': 'type_error.list'}]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__
    await self.app(scope, receive, send)
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 81, in handle
    response = await self._get_response_for_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 133, in _get_response_for_request
    return await self._call_handler_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 153, in _call_handler_function
    response_data, cleanup_group = await self._get_response_data(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 187, in _get_response_data
    parsed_kwargs = route_handler.signature_model.parse_values_from_connection_kwargs(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 196, in parse_values_from_connection_kwargs
    raise cls._create_exception(messages=messages, connection=connection) from e
litestar.exceptions.http_exceptions.ValidationException: 400: Validation failed for POST http://testserver.local/
Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_init_plugin.py", line 40, in _dec_pydantic_v1
    return model_type.parse_obj(value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 526, in parse_obj
    return cls(**obj)
           ^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Order
line_items
  value is not a valid list (type=type_error.list)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 190, in parse_values_from_connection_kwargs
    return convert(kwargs, cls, strict=False, dec_hook=deserializer, str_keys=True).to_dict()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 91, in _deserializer
    return decoder(target_type, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/contrib/pydantic/pydantic_init_plugin.py", line 42, in _dec_pydantic_v1
    raise ExtendedMsgSpecValidationError(errors=cast("list[dict[str, Any]]", e.errors())) from e
litestar._signature.types.ExtendedMsgSpecValidationError: [{'loc': ('line_items',), 'msg': 'value is not a valid list', 'type': 'type_error.list'}]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/middleware/exceptions/middleware.py", line 191, in __call__
    await self.app(scope, receive, send)
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 81, in handle
    response = await self._get_response_for_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 133, in _get_response_for_request
    return await self._call_handler_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 153, in _call_handler_function
    response_data, cleanup_group = await self._get_response_data(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/routes/http.py", line 187, in _get_response_data
    parsed_kwargs = route_handler.signature_model.parse_values_from_connection_kwargs(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thunder/work/litestar_mvce/litestar_venv/lib/python3.11/site-packages/litestar/_signature/model.py", line 196, in parse_values_from_connection_kwargs
    raise cls._create_exception(messages=messages, connection=connection) from e
litestar.exceptions.http_exceptions.ValidationException: 400: Validation failed for POST http://testserver.local/
{"status_code":400,"detail":"Validation failed for POST http://testserver.local/","extra":[{"message":"value is not a valid list","key":"line_items"}]}
INFO - 2023-11-08 10:15:48,994 - httpx - _client - HTTP Request: POST http://testserver.local/ "HTTP/1.1 400 Bad Request"
Traceback (most recent call last):
  File "/home/thunder/work/litestar_mvce/src/main.py", line 40, in <module>
    assert response.is_success
AssertionError

Litestar Version

2.3.2

Platform

• [X] Linux
• [ ] Mac
• [ ] Windows
• [ ] Other (Please specify in the description above)

---

[!NOTE]
While we are open for sponsoring on GitHub Sponsors and OpenCollective, we also utilize Polar.sh to engage in pledge-based sponsorship.

Check out all issues funded or available for funding on our Polar.sh Litestar dashboard

• If you would like to see an issue prioritized, make a pledge towards it!
• We receive the pledge once the issue is completed & verified
• This, along with engagement in the community, helps us know which features are a priority to our users.

[LonelyVikingMichael]

It seems that Litestar changed the way that multipart data is parsed.

from litestar._multipart import parse_multipart_form
from starlite.multipart import parse_multipart_form as starlite_parse_multipart_form

boundary = b"501688d00016004543513fb9e9784080"
body = (
    b'--501688d00016004543513fb9e9784080\r\nContent-Disposition: form-data; name="code"\r\n\r\n'
    b'123\r\n--501688d00016004543513fb9e9784080\r\nContent-Disposition: form-data; name="line_items"'
    b'\r\n\r\n[{"cost": 5, "description": "Not free food"}]\r\n--501688d00016004543513fb9e9784080\r\n'
    b'Content-Disposition: form-data; name="file"; filename="name.jpg"\r\nContent-Type: image/jpeg\r\n\r\n'
    b"todayisgood\r\n--501688d00016004543513fb9e9784080--\r\n"
)

litestar_result = parse_multipart_form(body=body, boundary=boundary)
starlite_result = starlite_parse_multipart_form(body=body, boundary=boundary)

print("starlite result: ", starlite_result)
print("litestar result: ", litestar_result)

assert litestar_result == starlite_result

Results:

# starlite result:  
{'code': 123, 'line_items': [{'cost': 5, 'description': 'Not free food'}], 'file': name.jpg - image/jpeg}

# litestar result:
{'code': '123', 'line_items': '[{"cost": 5, "description": "Not free food"}]', 'file': name.jpg - image/jpeg}

[LonelyVikingMichael]

2280

[guacs]

This was intentionally changed in the PR you linked, @LonelyVikingMichael. You can see a more detailed discussion regarding this here.

[LonelyVikingMichael]

This was intentionally changed in the PR you linked, @LonelyVikingMichael. You can see a more detailed discussion regarding this here.

Ah, thanks for the context. I am the person who was relying on the bug as a feature then

[provinzkraut]

@LonelyVikingMichael Have you tried explicitly declaring this as nested JSON in your Pydantic model with the Json type?

[LonelyVikingMichael]

@LonelyVikingMichael Have you tried explicitly declaring this as nested JSON in your Pydantic model with the Json type?

I've never used that type, you might be on to something. I've discussed internally with my work team and we've decided to take a different approach with mixed data types on HTTP POST.

I'll give this a shot if I ever get around to porting some of our other Starlite applications. Thank you all, happy to close this issue

[provinzkraut]

Thanks @LonelyVikingMichael!

I'll close this for now as the recommended approach for mixed (i.e. "nested" data) with multipart is explicitly declaring it.