bug: apiv1 get_replay failed to response in some cases

TrueRou commented 1 month ago

Describe the bug

Function api_get_replay may fail to response in some cases about the bad naming format in the header. We found that it always occurred when the username has characters that were not from ASCII charsets.

Exception: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 22-23: ordinal not in range(256)

Stacktrace:

ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
  |     yield
  |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 190, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 412, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    |     raise exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
    |     with collapse_excgroups():
    |   File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
    |     raise exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/osu-server/bancho.py/app/api/init_api.py", line 157, in http_middleware
    |     return await call_next(request)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
    |     raise app_exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
    |     with collapse_excgroups():
    |   File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    |     self.gen.throw(typ, value, traceback)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
    |     raise exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/osu-server/bancho.py/app/api/middlewares.py", line 22, in dispatch
    |     response = await call_next(request)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
    |     raise app_exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    |     raise exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
    |     await route.handle(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 557, in handle
    |     await self.app(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
    |     await route.handle(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
    |     await self.app(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    |     raise exc
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
    |     response = await func(request)
    |                ^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/fastapi/routing.py", line 299, in app
    |     raise e
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/fastapi/routing.py", line 294, in app
    |     raw_response = await run_endpoint_function(
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    |     return await dependant.call(**values)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/osu-server/bancho.py/app/api/v1/api.py", line 796, in api_get_replay
    |     return Response(
    |            ^^^^^^^^^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/responses.py", line 42, in __init__
    |     self.init_headers(headers)
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/responses.py", line 57, in init_headers
    |     raw_headers = [
    |                   ^
    |   File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/starlette/responses.py", line 58, in <listcomp>
    |     (k.lower().encode("latin-1"), v.encode("latin-1"))
    |                                   ^^^^^^^^^^^^^^^^^^^
    | UnicodeEncodeError: 'latin-1' codec can't encode characters in position 22-23: ordinal not in range(256)
    +------------------------------------

To Reproduce

Users who has unicode characters in their username uploaded scores.
Someone tried to download the replay by requesting /get_replay.
The exception threw with a bad replay file downloaded by someone.

Expected behavior

Refer to the starlette encoding issues: https://github.com/encode/starlette/pull/1163
Fix the problem.

bancho.py Version

5.2.2

Python Version

3.11.X (Default)

Relevant log output

No response

Additional context

No response

cmyui commented 1 month ago

Ah we recently had this problem on Akatsuki. The issue is the content-disposition header in HTTP only supports latin-1 content, however osu! beatmap song names may contain unicode characters.

We fixed it here: https://github.com/osuAkatsuki/score-service/commit/fa116d81466ab76ad9d074212faf65dfb866b15e

This aligns replay names with those from bancho.

If anyone would be interested in taking a crack at this, it should be pretty straightforward!

anirudh1117 commented 1 month ago

Hi @TrueRou , can i start working on this ? i have go through the api.py file and controller "api_get_replay" so username may contain unicode characters, you want to implement something like this ?

f"attachment; filename*=utf-8''{quote(username)} - ".encode('latin-1')

osuAkatsuki / bancho.py