[Tagging] Perform load tests for copying tags when blocks are copied

[yusuf-musleh]

In relation to https://github.com/openedx/modular-learning/issues/165, we need to perform some load testing to see how xblocks that have a lot of tags applied would react when copied/pasted after the changes have been implemented and the tags are included in the process.

Depending on the performance of copying, we would need to look into ways to perform the copying over of tags asynchronously if it is actually significantly slower.

[yusuf-musleh]

I've done a couple of performance load tests with different scenarios to see the impact of having tags with copying/pasting/deleting units. This is mainly to see how unit related actions perform when tags are included in the picture, whether we need to make optimizations or turn certain actions async.

Unit with 14k tags, 15 components

For the first set of tests I've applied 14k tags to a unit that contains 15 blocks, and for the next set of tests I've applied 14k tags to each of the 15 blocks. Here are the different tests:

Copying Units

Copying a unit that has 14k tags applied is still completed under 1s, however with the significantly large number of tags shoots up to over 9s to complete.

Pasting Units

Pasting of a unit that contains 14k tags, either within the same section or in a different section (both in the same course). This was a bit strange, since sometimes it would take ~35s to complete and sometime it would shoot up to ~2.6min to complete. It seems like the more units that have a large number of tags in the section, the longer it takes to paste it, however sometime this wasn't the case, but my assumption is that it's likely related.

Pasting of a unit that contains 14k + (15 x 14k) tags applied. These ones took significantly more time time (notice response time in minutes) as expected. In some attempts, the paste would just hang (no errors, and the request is still waiting in the browser), I let it run for +30min with nothing, however when I refreshed the course was pasted successfully. Something interesting I also noticed, when pasting the unit in a different course that belongs to another org, it takes much longer to complete, even though there are less tags carried over (since a bunch of the tags belong only to the first course's org)

Deleting Units

Deleting the unit takes some time with a large number of tags, but seems to decrease as the number of units decreases.

Thoughts

These are tests done on a very (ridiculously) large number tags, probably unlikely to happen in a real world scenario, especially since there is currently a limit (of 100) set on the number of tags applied per-item. But this gives us a general idea of how slower the response times get. We notice, that if there is a course that has many units, each with many blocks, and all of them have many tags applied, all this has an impact on the response times for the different interactions/actions that can be done on the units/blocks.

Below are performance tests within the set limits.

Unit with 30 components, 100 tags each

Copying Unit

Copying remained under 500ms which is pretty fast, no major concerns there.

Pasting Unit

Pasting actually takes some time, averaging ~35s across the different scenarios, much better than the previous ~2min. I'm not sure if it crosses the threshold to covert it to an async task, but might be worth digging further and looking into ways to optimize this. For reference here is pasting the same unit with no tags:

We can see that when the tags are included, it accounts for the bulk of the pasting action.

Deleting Unit

Deleting averaged out to between 1-2s, not too bad, especially since there is an indication that is is deleting.

Thoughts

Pasting is the heaviest of all 3 operations, but we can definitely look into optimizing how tags are added to the newly pasted unit, since they account for the bulk of the time. Since the number of applied tags is capped, the response time is not expected to reach the durations seen in the earlier (uncapped) tests, so it seems like the better route is to optimize the pasting action rather than going the async route.

[bradenmacdonald]

Thanks for these findings @yusuf-musleh. I agree that it seems like there is something quite inefficient about how tags are currently pasted, and we need to create a new ticket for optimizing that as a follow-up ticket.

[yusuf-musleh]

@bradenmacdonald Agreed, I created https://github.com/openedx/modular-learning/issues/208 as a follow up ticket.

[yusuf-musleh]

After a bit more digging, it turns out that the the culprit was the django-debug-toolbar. I commented it out from the settings, and that dropped the time significantly. It was adding 6-9s additional seconds to the request. With the debug tool bar disabled, I did a few load tests for pasting with/without tags since the previous load tests definitely were impacted by it and here are the updated results:

It's a lot more reasonable now, units with 30*100 tags taking 1-4s to paste, and units with no tags taking less than a second. After discussing with @bradenmacdonald, we concluded that no further optimization is currently needed.