search

overhangio / openedx-scorm-xblock

SCORM XBlock for Open edX
GNU Affero General Public License v3.0
37 stars 48 forks source link

very slow uploads when using the S3 backend #14

Closed sampaccoud closed 3 years ago

sampaccoud commented 3 years ago

Our users reported timeouts when uploading Scorm modules. We had to drastically increase timeouts on our web servers to solve the issue but, digging a little bit, we think uploads are abnormally slow.

On first glance, the sequence of events happening while the user is holding the request is:

This seems far too many things to do while holding on a user request. It could be improved:

  1. upload the archive to a temp file or memory in the first place so it can be unzipped directly

or

  1. upload the zip file to S3 in the request and subcontract the unzipping to Celery asynchronously

Even more efficient would be to upload directly to S3 with an upload policy... and the unzipping could be done directly in S3 via a lambda trigger but it would not work for alternative S3 hosting, swift, etc. so I think the two first solutions are better for the OpenEdX community.

regisb commented 3 years ago

I think your analysis of the problem is correct @sampaccoud. Can you just specify which version of the xblock you are running?

EDIT: do you have an opinion on whether it's really useful to store the zip file on S3? My feeling is that we could get rid of it entirely.

sampaccoud commented 3 years ago

We are running openedx-scorm-xblock==9.2.5 on Ironwood. Yes, we also feel that the zip file should be trashed after being unzipped...

regisb commented 3 years ago

Solved in v10.1.0 and v9.3.0.