search

learningequality / ka-lite

KA Lite: lightweight web server for serving core Khan Academy content (videos and exercises) without needing internet connectivity
https://learningequality.org/ka-lite/
Other
456 stars 306 forks source link

Make video/thumbnail download more robust #5536

Closed benjaoming closed 6 years ago

benjaoming commented 6 years ago

Summary

Debugging #5528, I found that the issue seems to be the lack of retries when connections are broken or temporarily unresolved.

Any error in downloading a video or thumbnail results in an item being skipped in the queue. This could also explain why the resized videos torrent set has consistently been missing a couple of items.. downloading 30+k files would usually result in a couple of errors. From my own connection, I had 197 errors out of 15k thumbnail downloads. FYI: @jamalex

TODO

If not all TODOs are marked, this PR is considered WIP (work in progress)

Reviewer guidance

Assuming no external use of download_file and other refactored components. Central server will not be upgraded immediately.

Issues addressed

5528

codecov[bot] commented 6 years ago

Codecov Report

Merging #5536 into 0.17.x will increase coverage by 2.15%. The diff coverage is 85.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           0.17.x    #5536      +/-   ##
==========================================
+ Coverage   62.59%   64.75%   +2.15%     
==========================================
  Files         117      117              
  Lines        6556     6608      +52     
==========================================
+ Hits         4104     4279     +175     
+ Misses       2452     2329     -123
Impacted Files Coverage Δ
kalite/topic_tools/content_models.py 78.08% <ø> (+2.8%) :arrow_up:
kalite/updates/download_track.py 78.57% <ø> (+19.04%) :arrow_up:
kalite/updates/settings.py 100% <100%> (ø) :arrow_up:
...alite/updates/management/commands/videodownload.py 58.82% <60%> (+58.82%) :arrow_up:
kalite/updates/videos.py 85.93% <87.93%> (+32.6%) :arrow_up:
kalite/main/models.py 86.51% <0%> (+1.4%) :arrow_up:
kalite/facility/api_resources.py 95.2% <0%> (+1.54%) :arrow_up:
kalite/facility/api_views.py 64% <0%> (+10%) :arrow_up:
kalite/topic_tools/base.py 57.57% <0%> (+12.12%) :arrow_up:
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5b2bf2e...36c66ae. Read the comment docs.