openzim / python-scraperlib

Collection of Python code to re-use across Python-based scrapers
GNU General Public License v3.0
19 stars 16 forks source link

YoutubeDownloader context manager option doesn't achieve parallelism #61

Closed satyamtg closed 4 years ago

satyamtg commented 4 years ago

The current way in which the context manager is implemented along with download() in YoutubeDownloader, it makes it necessary to send parallel requests to download() in order to achieve parallelism, otherwise it runs in serial mode even with more number of workers (as the download() inherently waits for the executor job to finish before returning. See https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.exception)

We can have an option in download() to specify whether we want to wait or not