search

overhangio / openedx-scorm-xblock

SCORM XBlock for Open edX
GNU Affero General Public License v3.0
37 stars 46 forks source link

Abnormal S3 file upload speed of 30s/MB is causing timeouts #42

Closed dyudyunov closed 5 months ago

dyudyunov commented 1 year ago

When using the S3 backend for the file storage I discovered that files >1MB are failing to upload.

Backend logs have the 499 error(client closed the connection before response):

==> /edx/var/log/nginx/access.log <==
https [07/Jun/2023:12:48:17 +0000]  "POST /xblock/block-v1:scorm_rga+scorm_rga+scorm_rga+type@scorm+block@3f281ac8d2a644dc99aa38b2fd052f9b/handler/studio_submit HTTP/1.1" 499 0 60.193 "https://cms.sandbox.com/container/block-v1:scorm_rga+scorm_rga+scorm_rga+type@vertical+block@562359cf5b6b47c1a33abceecc4a5588" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" 127.0.0.1:18010 59.960
https [07/Jun/2023:12:48:17 +0000]  "POST /xblock/block-v1:scorm_rga+scorm_rga+scorm_rga+type@scorm+block@3f281ac8d2a644dc99aa38b2fd052f9b/handler/studio_submit HTTP/1.0" 499 0 59.959 "https://cms.sandbox.com/container/block-v1:scorm_rga+scorm_rga+scorm_rga+type@vertical+block@562359cf5b6b47c1a33abceecc4a5588" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" 127.0.0.1:8010 59.956

The browser request ended up with the 504 (timeout)

After the investigation, I've found that increasing the NGINX timeouts fixes the issue, but it doesn't look like a real solution.

Here are some measurements for my test files (time for the studio_submit request to be done):

File Size Response Time
360 KB 11.15 sec
2.7 MB 80 sec

Note: FileSystem storage has no such issues

regisb commented 11 months ago

Hi @dyudyunov! Sorry about the long overdue response.

I see that you are sharing nginx access logs, so it means that you are not using Tutor to run Open edX. Is that correct? I also see that your logs are in /edx/var/log/nginx/access.log, which seems to indicate that you are running the native installation -- right?

Still, the response times you are reporting are clearly abnormal. @naumanshafi as the author of the initial implementation of the S3 backend, did you also observe 30s/MB upload times?

dyudyunov commented 11 months ago

I see that you are sharing nginx access logs, so it means that you are not using Tutor to run Open edX. Is that correct? I also see that your logs are in /edx/var/log/nginx/access.log, which seems to indicate that you are running the native installation -- right?

That's correct

ziafazal commented 11 months ago

@dyudyunov response time depend on the internet upload speed too. Could you please share some stats about you internet upload speed?

@regisb S3 backend implementation @naumanshafi did was for overriding default S3 backend to allow storage and retrieval of content from private S3 bucket. @dyudyunov are you setting this "STORAGE_FUNC": "openedxscorm.storage.s3"?

dyudyunov commented 11 months ago

@dyudyunov response time depend on the internet upload speed too. Could you please share some stats about you internet upload speed?

This is definitely not the root cause. Tested different people in several locations with several connection options. Here is my speedtest result: 

Testing download speed................................................................................
Download: 197.49 Mbit/s
Testing upload speed......................................................................................................
Upload: 78.85 Mbit/s

@dyudyunov are you setting this "STORAGE_FUNC": "openedxscorm.storage.s3"?

Yes, no difference

regisb commented 8 months ago

@dyudyunov the issue should not appear again in Quince (tutor v17.0.0). Can you confirm?

dyudyunov commented 8 months ago

@dyudyunov the issue should not appear again in Quince (tutor v17.0.0). Can you confirm?

can't guarantee I can make it till 2024 🙂

regisb commented 8 months ago

I'm not sure I understand -- Quince was released yesterday, and so you should be able to test today. No need to wait until 2024 :)

dyudyunov commented 8 months ago

I meant that I have high-priority tasks to do and I'm not sure I will switch to this one soon)

regisb commented 8 months ago

Right! I did not understand that :sweat_smile: We'll try to do the testing on our side then.

Abdul-Muqadim-Arbisoft commented 6 months ago

I recently uploaded a SCORM package, sized at 6.2 MB, and it took approximately 67 seconds to complete. My network speeds at the time were as follows: download speed at 14.5 Mbps and upload speed at 18.5 Mbps.

DawoudSheraz commented 5 months ago

The related PR (https://github.com/overhangio/openedx-scorm-xblock/pull/46) has been closed as won't do, closing the issue as well.