Better youtube_dl requests mgmt

rgaudin commented 3 years ago

The new structure from #119 PR accidentally strengthen something wrong in our requests management. Every youtube_dl call leads to a request to the video webpage from which youtube_dl extracts the information it needs to then

download the video file(s)
download the thumbnail(s)
download the subtitle(s) As this webpage is subject to the quota, this is not very efficient as we'd call it 3 times in case we have nothing for that video in S3.

We should also probably save those subtitles in S3 although those can change over time. As for thumbnails, I don't think this matters much until users raise concern about subtitles being outdated (then we'd expire them after some time).

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

benoit74 commented 1 month ago

Need to be done together with https://github.com/openzim/youtube/issues/177

And caching S3 subtitles is now tracked as #277

openzim / youtube

Better youtube_dl requests mgmt #122