search

jazzband / django-simple-history

Store model history and view/revert changes from admin site.
https://django-simple-history.readthedocs.org
BSD 3-Clause "New" or "Revised" License
2.11k stars 464 forks source link

HistoricForeignKey is very slow #1325

Open gabn88 opened 4 weeks ago

gabn88 commented 4 weeks ago

Describe the bug We have tens of thousands of instances (not millions), but the historic foreignkey loads already very slow (+60 seconds)

To Reproduce Create a Model (in our case with 2 historicforeignkeys). Load a few thousand Models and RelatedModels. Load the historic Model instance. Load the related model. Sometimes it takes a few seconds, but it could also take a full minute.

Expected behavior It should load within a few ms.

Environment (please complete the following information):

Additional context I think the reason is the number of historical items. We have two historicalforeignkeys on the Model. The one with +/- 30.000 historic rows takes about 5-6 seconds and the one with +/- 600.000 historic rows takes 60+ seconds.

gabn88 commented 4 weeks ago

After digging some more into this, it seems that the HistoricForeignKey first creates a queryset of all possible historic instances (all 600.000) in HistoricForwardManyToOneDescriptor and then filters it down. This seems to be the root cause.

For our model the pk relation will stay the same, so I can take a shortcut -> select the pk and then get the historical object. This reduced the time to get the historic instance from 60+ seconds to a few milliseconds.

tim-schilling commented 3 weeks ago

@gabn88 thanks for the report! Do you have the time to go the extra step of creating a forkable project that would allow us to reproduce this?