FAForever / server

The servercode for the Forged Alliance Forever lobby
http://www.faforever.com
GNU General Public License v3.0
66 stars 61 forks source link

Add config option to specify json float serialization precision #905

Open Gatsik opened 2 years ago

Gatsik commented 2 years ago

Closes #875

Gatsik commented 2 years ago

test_ladder_game_draw_bug and test_game_ended_broadcasts_rating_update are failing probably because of customized precision Do you want this custom precision only for queue_pop_time_delta field in matchamker_info message?

Askaholic commented 2 years ago

Hey, nice work! I think it makes sense to use the same precision for everything, so those tests probably need to be adjusted to have higher tolerance for error.

The second one is interesting because it means there will be some edge cases where the rating change is so small that it doesn’t appear to change at all. I think that’s ok though since the full change history will still be available through the replay details page and the rating graph.

Askaholic commented 2 years ago

One thing we have to keep in mind is that if we’re messing with the json encoding, we need to be very careful of the performance impact since this is one of the hottest functions in the entire code base. It would be amazing if we could just override the float serialization code and leave the rest of the implementation untouched. I found a little snippet that does this although it uses the mock.patch functionality to override private functions from the Json library. Still, this may be a good source of inspiration:

https://gist.github.com/Sukonnik-Illia/ed9b2bec1821cad437d1b8adb17406a3

Gatsik commented 2 years ago

It looks like we can't do much without creating custom python package that will use compiled C code. I did some comparison between different methods using slightly modified script taken from https://developpaper.com/speed-comparison-of-five-json-libraries-in-python/ and here are the results of encoding on my machine:

patching (example given above): 107.800
current (this pr's code):  28.629

Default serialization

simplejson: 6.981
ujson: 3.503
stdlib json: 4.500

Round floats approach (https://stackoverflow.com/a/53798633):

simplejson: 14.196
ujson: 10.481
stdlib json: 11.109

Overriding iterencode function, which contains floatstr method, directly without patching:

15.991

Compiling custom mix of json and simplejson (most of the code is a copy from https://github.com/python/cpython/pull/13233/, built with setup.py from simplejson):

6.589

Time is in seconds, the data for encoding is:

{
    "command": "game_info",
    "visibility": "public",
    "password_protected": True,
    "uid": 13,
    "title": "someone's game",
    "state": "playing",
    "game_type": "custom",
    "featured_mod": "faf",
    "sim_mods": {},
    "mapname": "scmp_009",
    "map_file_path": "maps/scmp_009.zip",
    "host": "Foo",
    "num_players": 2,
    "launched_at": 1111111111.1112312312312,
    "rating_type": "faf",
    "rating_min": None,
    "rating_max": None,
    "enforce_rating_range": False,
    "team_ids": [
        {
            "team_id": 1,
            "player_ids": [1],
        },
        {
            "team_id": 2,
            "player_ids": [2],
        },
    ],
    "teams": {
        1: ["Foo"],
        2: ["Bar"],
    },
}

There is also a package called orjson which is super fast, but requires dict keys to be strings, so with modified initial data, in which all keys are strings, the result is:

0.624

Otherwise, using "round floats" approach in which we also convert dict keys to str:

8.457