alexta69 / metube

Self-hosted YouTube downloader (web UI for youtube-dl / yt-dlp)
GNU Affero General Public License v3.0
7.04k stars 451 forks source link

too many open files error, makes metube selfterminate #485

Open andbir opened 3 months ago

andbir commented 3 months ago

If you download a very long playlist, in my case the DUST sci-fi archives meTube v1.240620 running on QNAP x64 will eventually choke on itself, thus not be able to handle its' own database calls and exit: n = pdAdSu3kY1sHV-U8 ; player = https://www.youtube.com/s/player/d2e656ee/player_ias.vflset/en_US/base.js WARNING: [youtube] CeloPsVCZ3A: nsig extraction failed: You may experience throttling for some formats n = hQ7WQeIieuhkyUsa ; player = https://www.youtube.com/s/player/d2e656ee/player_ias.vflset/en_US/base.js WARNING: [youtube] Failed to download m3u8 information: <urllib3.connection.HTTPSConnection object at 0x7fc3c67a4ed0>: Failed to establish a new connection: [Errno 24] Too many open files ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-3' coro=<DownloadQueue.import_queue() done, defined at /share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py:213> exception=error("db file doesn't exist; use 'c' or 'n' flag to create a new db")> Traceback (most recent call last): File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 215, in __import_queue File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 317, in add File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 295, in add_entry File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 189, in put File "/opt/Apache83/lib/python3.11/shelve.py", line 243, in open File "/opt/Apache83/lib/python3.11/shelve.py", line 227, in init File "/opt/Apache83/lib/python3.11/dbm/init.py", line 85, in open dbm.error: db file doesn't exist; use 'c' or 'n' flag to create a new db INFO:ytdl:downloading Sci-Fi Short Film “Sleepworking” | DUST ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-2' coro=<DownloadQueue.download() done, defined at /share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py:366> exception=OSError(24, 'Too many open files')> Traceback (most recent call last): File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 374, in download File "/share/CACHEDEV3_DATA/.qpkg/meTube83/metube/app/ytdl.py", line 108, in start File "/opt/Apache83/lib/python3.11/multiprocessing/process.py", line 121, in start File "/opt/Apache83/lib/python3.11/multiprocessing/context.py", line 224, in _Popen File "/opt/Apache83/lib/python3.11/multiprocessing/context.py", line 281, in _Popen File "/opt/Apache83/lib/python3.11/multiprocessing/popen_fork.py", line 19, in init File "/opt/Apache83/lib/python3.11/multiprocessing/popen_fork.py", line 65, in _launch OSError: [Errno 24] Too many open files

PikuZheng commented 3 months ago

A known Linux kernel bug causes this error in Linux kernel 4.19-5.9, maybe it's the reason of your issue?