search

spotDL / spotify-downloader

Download your Spotify playlists and songs along with album art and metadata (from YouTube if a match is found).
https://spotdl.readthedocs.io/en/latest/
MIT License
17.74k stars 1.62k forks source link

OSError: Too Many Files, when downloading a bigger playlist #1978

Open mdawood94 opened 11 months ago

mdawood94 commented 11 months ago

System OS

MacOS

Python Version

3.11 (CPython)

Install Source

pip / PyPi

Install version / commit hash

v4.2.4

Expected Behavior vs Actual Behavior

When downloading a larger playlist after about 50-70 tracks have been downloaded the terminal throws an error saying Too many open files, and all subsequent tracks do not get downloaded and it starts saying audio provider error.

Screenshot 2023-12-09 at 2 45 31 PM

Steps to reproduce - Ensure to include actual links!

Simple spotdl command on any large playlist should show this issues

Traceback

OSError: [Errno 24] Too many open files
OSError: [Errno 24] Too many open files
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=ZiZWzVfeWVg
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=Aduc9vE3Mog
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=_GfaWmDDpNM
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=XHA8OP2IbXU

Other details

No response

neverexpecthings commented 11 months ago

gretting, since you're using mac, please tell me the value of "open file" using ulimit -a

jaydenrasmussen commented 11 months ago

default number of open file descriptors on mac is 256. This behavior does not goes away when using the --threads flag

❯ ulimit -n
256
tutozz commented 11 months ago

this tutorial solved temporarily the issue: https://gist.github.com/skylock/0117ec637d468f91260927b43b816eda

but closing the file after saving it should solve this issue (file stay open in RAM even after song finished)

jaydenrasmussen commented 11 months ago

looking through the source quickly it looks like the project follows the pythonic with in a few different places but doesn't explicitly call os.close which I think is the source of the issue ref: https://github.com/spotDL/spotify-downloader/blob/f90cfb3f5ab1b15e73d0393d531908d4ac789e7d/spotdl/console/save.py#L87-88

hofleverancier commented 10 months ago

Confirming this issue on Linux (using Ubuntu 23). This starts to happen after downloading a couple hundred files. Max open files on my OS is not altered, so should be whatever the default is.

Downloaded "Le Roy - See the Light Ahead": https://music.youtube.com/watch?v=Kd5reUZaGEI                                
OSError: [Errno 24] Too many open files                                                                                 
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=aEV83SlKyU0                               
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'                                                                  
OSError: [Errno 24] Too many open files: 'cookies.txt'
antoinedelia commented 10 months ago

I'm having the same issue when downloading a lot of files from WSL using Ubuntu.

For reference, I'm using the command: spotdl download saved --user-auth --lyrics musixmatch --output "{album-artist}/{album-artist} - {title}"

Downloaded "ILLENIUM - Where'd U Go (Thoreau Rework)": https://music.youtube.com/watch?v=ZlTI54Bwe1E
OSError: [Errno 24] Too many open files
Downloaded "The Him - Broken Love (feat. Parson James)": https://music.youtube.com/watch?v=UMygH95XZqY
OSError: [Errno 24] Too many open files
AudioProviderError: YT-DLP download error - https://music.youtube.com/watch?v=lRJnVT_OPhk
Couldn't read cache at: /home/antoinedelia/.spotdl/.spotipy
OSError: [Errno 24] Too many open files
Couldn't read cache at: /home/antoinedelia/.spotdl/.spotipy
OSError: [Errno 24] Too many open files
Couldn't read cache at: /home/antoinedelia/.spotdl/.spotipy
OSError: [Errno 24] Too many open files
Couldn't read cache at: /home/antoinedelia/.spotdl/.spotipy
OSError: [Errno 24] Too many open files
Couldn't read cache at: /home/antoinedelia/.spotdl/.spotipy
OSError: [Errno 24] Too many open files

When I'm running ulimit -n, I got 1024.

xnetcat commented 10 months ago

try running ulimit -S 4096

I will investigate this more thoroughly whenever I get some time

isle9 commented 8 months ago

Still have this issue on Ubuntu 23.10.

jstebenne commented 6 months ago

I have the same issue on Ubuntu 22.04

OstlerDev commented 6 months ago

I was able to fix it by using the docker container, since I believe the Docker VM is setup with a much higher "open file" limit. I tested it on a 1300 song playlist and it was able to download ~400 songs before erroring out. The code definitely still needs a fix to close files after it is done with with them, but as a temporary workaround you can use the docker container and run it a few times with the "sync" command.

docker run --rm -v $(pwd):/music spotdl/spotify-downloader sync https://open.spotify.com/playlist/playlistURL --save-file playlist.spotdl --format mp3 --threads 8
william-green commented 4 months ago

I am experiencing this problem on Ubuntu. OSError: [Errno 24] Too many open files

Screenshot from 2024-07-12 21-25-45

As someone else suggested, I'm thinking the file handles are not being closed. If the code is recursive or the file handle isn't explicitly closed then it could cause this error. I have yet to look at the source code.

I wrote a little bash script to kill and restart every 30 mins with a crontab.

luna-null commented 3 months ago

I'm getting this error, too. I believe this may have something to do with spotipy. Something is preventing requests that the program opens to get data about the song from closing. I tried adding os.close() everywhere, and that didn't work. I also tried doing with requests.get() as session: statements for every requests.get() and requests.Session mention, and that didn't work either.

                       Traceback (most recent call last):                                                                                      
                         File "spotdl/download/downloader.py", line 538, in search_and_download                                                
                         File "spotdl/utils/search.py", line 540, in reinit_song                                                               
                         File "spotdl/types/song.py", line 100, in from_url                                                                    
                         File "spotipy/client.py", line 461, in album                                                                          
                         File "spotdl/utils/spotify.py", line 199, in _get                                                                     
                         File "spotdl/utils/spotify.py", line 195, in _get                                                                     
                         File "spotipy/client.py", line 247, in _internal_call                                                                 
                         File "spotipy/client.py", line 238, in _auth_headers                                                                  
                         File "spotipy/oauth2.py", line 238, in get_access_token                                                               
                         File "spotipy/oauth2.py", line 257, in _request_access_token                                                          
                         File "requests/sessions.py", line 637, in post                                                                        
                         File "requests/sessions.py", line 589, in request                                                                     
                         File "requests/sessions.py", line 703, in send                                                                        
                         File "requests/adapters.py", line 519, in send                                                                        
                       requests.exceptions.ConnectionError: HTTPSConnectionPool(host='accounts.spotify.com', port=443):                        
                       Max retries exceeded with url: /api/token (Caused by                                                                    
                       NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x74bfd760e6c0>: Failed to                            
                       establish a new connection: [Errno 24] Too many open files'))

This is an example of one of the main error messages I'd get.

easter-eat-ser commented 1 month ago

Still having this issue, file limit is 1024 but it errors at around 200 songs. The error is slightly different, instead saying:

ConnectionError: HTTPSConnectionPool(host='accounts.spotify.com', port=443): Max retries exceeded with url:        
/api/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x725d6e8f6360>: Failed to 
establish a new connection: [Errno 16] Device or resource busy'))                                                  
Couldn't read cache at: /home/easter/.spotdl/.spotipy
Couldn't read cache at: /home/easter/.spotdl/.spotipy
Couldn't read cache at: /home/easter/.spotdl/.spotipy

(The too many open files error also occurs) I downloaded it from the Arch AUR. Does this have anything to do with the problem?

symonty commented 1 month ago

Getting same error with large lists, is there a way to limit open files? I agree about open files not closing, also noted the sockets are not closing and when I investigate the total open sockets it is far longer than then present or recently completed DL.

tcp4 0 0 10.0.1.4.64872 74.125.157.8.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64868 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64865 71.33.34.14.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64864 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64863 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64856 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64855 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64846 74.125.157.134.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64843 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64834 173.194.56.103.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64831 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64829 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64825 172.217.14.206.443 CLOSE_WAIT tcp4 24 0 10.0.1.4.64823 5.79.76.225.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64796 173.194.166.232.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64793 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64789 173.194.12.170.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64786 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64784 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64781 173.194.8.137.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64778 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64777 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64771 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64767 74.125.157.40.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64764 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64762 173.194.8.168.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64759 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64757 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64753 173.194.12.137.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64750 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64748 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64746 173.194.8.102.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64742 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64739 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64736 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64732 173.194.167.70.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64729 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64728 173.194.8.102.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64723 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64722 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64720 173.194.8.135.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64716 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64713 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64711 74.125.103.104.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64707 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64704 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64701 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64674 74.125.170.6.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64671 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64668 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64665 173.194.8.74.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64661 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64657 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64654 173.194.56.201.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64652 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64643 71.33.34.15.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64641 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64639 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64637 74.125.170.7.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64632 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64630 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64627 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64590 173.194.167.103.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64587 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64582 74.125.157.6.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64578 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64577 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64573 74.125.157.42.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64570 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64569 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64565 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64560 74.125.103.168.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64557 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64556 173.194.167.230.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64551 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64548 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64545 173.194.166.232.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64542 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64541 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64539 173.194.8.198.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64534 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64532 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64525 74.125.5.169.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64523 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64522 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64519 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64494 71.33.34.14.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64493 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64492 173.194.8.138.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64489 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64488 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64484 71.33.34.12.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64482 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64481 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64480 173.194.56.170.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64475 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64474 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64469 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64465 173.194.167.70.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64461 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64459 74.125.103.199.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64455 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64454 173.194.12.137.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64451 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64450 74.125.103.168.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64446 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64445 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64435 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64434 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64433 172.217.14.206.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64410 173.194.8.41.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64407 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64401 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64382 173.194.167.230.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64379 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64375 173.194.8.136.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64372 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64371 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64370 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64364 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64363 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64357 173.194.12.202.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64352 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64346 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64342 172.217.14.238.443 CLOSE_WAIT tcp4 0 0 10.0.1.4.64341 172.217.14.238.443 CLOSE_WAIT

Over 230 of these in CLOSE_WAIT

symonty commented 1 month ago

I do think this is about open sockets

jimmyprior commented 4 weeks ago

See #2201 for temporary solution