open5e / open5e-api

The api for open5e.com
Other
156 stars 97 forks source link

Eager Loading for CreatureViewSet #602

Closed calumbell closed 2 days ago

calumbell commented 5 days ago

This PR goes part toward resolving #577 by starting to address the N+1 problem on the /creatures endpoint.

This content builds on @tylereed 's work in PR #592, only this time for the /creatures endpoint. One notable addition I made is the action parameter passes to setup_eager_loading method, which allows us to configure different eager loading options for list views (ie. v2/creatures) and single views (ie. v2/srd_goblin).

Follow ups

With the current configuration of the CreatureViewSet, it is possible to dynamically select a serializer based on whether the user has request ONE or MANY creatures. It might be a good idea to set these up to return sensible default fields for different contexts, as the Creature model is quite large.

Tests

Tests were conducted using Django Debug Tools

/creatures without fields

For endpoint v2/creatures/?limit=50, which fetches the first 50 creatures in the dataset. This endpoint currently massively overfetches for the purposes of the /monsters page on the front-end

Staging, time (ms) Staging, SQL Queries Feature, time (ms) Feature, SQL Queries
1st 4935.58 1127 3779.07 683
2nd 4788.69 "" 3773.86 ""
3rd 4839.77 "" 3679.95 ""
4th 4856.14 "" 3679.65 ""
Average 4855.045 3728.1325

Reduction: 1.126s (4 s.f.) Percent Reduction: 76.79% (4 s.f.)

The N+1 problem still very much exists when the data is fetched this way. The problematic fields are CreatureAction and CreatureAbility. These fields are still being actively worked on, so it feels a bit premature to begin optimising them. I am pointing this out because there is definitely a lot of scope to reduce these response times further.

/creatures with fields

For endpoint v2/creatures/?fields=name,document,key,size,challenge_rating_decimal,challenge_rating_text&document__fields=name,key&size__fields=name,key&limit=50 which returns data in the following format (everything the /monsters page requires to fully function)

{
  "document": {
     "key": "bfrd",
      "name": "Black Flag SRD"
   },
  "key": "bfrd_aboleth",
  "name": "Aboleth",
  "size": {
    "key": "large",
    "name": "Large"
   },
   "challenge_rating_decimal": "10.000",
   "challenge_rating_text": "10"
 },
Staging, time (ms) Staging, SQL Queries Feature, time (ms) Feature, SQL Queries
1st 269.63 104 175.89 10
2nd 258.11 "" 174.83 ""
3rd 261.96 "" 176.4 ""
4th 264.43 "" 171.37 ""
5th 263.89 "" 172.64 ""
Average 263.604 174.226

Reduction (ms): 89.378 ms Reduction (%): 66.094% (4 s.f.)

Omitting all fields except those that would be required by , it looks like we can reduce response times by a third.