osuAkatsuki / bancho.py

An osu! server for the generic public, optimized for maintainability in modern python
https://akatsuki.gg
MIT License
201 stars 125 forks source link

misc: determine a method to globally handle custom json serialization using `httpx` #586

Open cmyui opened 4 months ago

cmyui commented 4 months ago

At the moment, our server has a couple of locations where objects of types such as datetime are not able to be serialized into JSON due to a lack of support in the stdlib json module. httpx -- the http client we use for outgoing requests, relies specifically on the stdlib json module.

I see a couple of ways we can tackle this:

  1. Use post(headers={'Content-Type': 'application/json'}, data=custom_json_dumps(...)) in each call. Never use post(json=...). Seems quite error prone/duplicative.
  2. Subclass AsyncClient, override the build_request method, copy most from httpx but use a custom Request which overrides __init__ to perform option 1's suggestion automatically before yielding to super().__init__.
  3. Monkey-patch httpx._content.encode_json to provide a custom JSONEncoder subclass to json.dumps(cls=...).
  4. Implement the httpx maintainer's proposal to httpx to implement support for AsyncClient(request_class=..., response_class=...). Do the same __init__ override as suggested in option 1.

My preference is probably option 4 -- seems the best for the overall ecosystem. We could potentially do one of the other options in the short term as I believe this is currently having an effect on production.

cmyui commented 4 months ago

cc: @tsunyoku @NiceAesth @minisbett

minisbett commented 4 months ago

potential hotfix:

import orjson
import httpx._content

from typing import Any, Tuple, Dict

def encode_json(json: Any) -> Tuple[Dict[str, str], httpx._content.ByteStream]:
    body = orjson.dumps(json)
    content_length = str(len(body))
    content_type = "application/json"
    headers = {"Content-Length": content_length, "Content-Type": content_type}
    return headers, httpx._content.ByteStream(body)

httpx._content.encode_json = encode_json