search

spotDL / spotify-downloader

Download your Spotify playlists and songs along with album art and metadata (from YouTube if a match is found).
https://spotdl.readthedocs.io/en/latest/
MIT License
15.55k stars 1.48k forks source link

spotdl fails due to large number of opened sockets in CLOSE_WAIT state #2101

Open lpryszcz opened 1 month ago

lpryszcz commented 1 month ago

System OS

Linux

Python Version

3.10 (CPython)

Install Source

pip / PyPi

Install version / commit hash

v4.2.5

Expected Behavior vs Actual Behavior

Hi, first of all, thanks for this wonderful program! spotdl crashed a few times in my hand, most likely due to very large number of opened sockets in CLOSE_WAIT state. Could unused socket connections be closed once they're not needed? Below you can see how the number of list of open files for spotdl is growing when downloading a playlist with 2k entries. spotdl --threads 6 --overwrite skip --m3u ../LikedSongs.m3u URL

$ for i in `seq 1 1 100`; do echo `date` `lsof |grep -c spotdl`; sleep 1m; done
Mon 20 May 14:30:36 CEST 2024 32
Mon 20 May 14:31:42 CEST 2024 238
Mon 20 May 14:32:48 CEST 2024 960
Mon 20 May 14:33:54 CEST 2024 1304
...
Mon 20 May 15:04:30 CEST 2024 12511
Mon 20 May 15:05:40 CEST 2024 12741

I observed over 12k opened files/connections after downloading ~1.2k entries. Solution for me was to set sufficiently large ulimit -n - it has to be ~10x larger than the number of entries in the playlist.

Cheers,

Steps to reproduce - Ensure to include actual links!

  1. Start monitoring number of opened files lsof | grep spotdl
  2. Start downloading a large playlist
  3. spotdl will crash if ulimit -n isn't sufficiently large

Traceback

...
Downloaded "Major Lazer - Lean On (feat. MØ & DJ Snake)": https://music.youtube.com/watch?v=0mjiJ4JlW1g                                     
Downloaded "Major Lazer - Get Free": https://music.youtube.com/watch?v=A67ia5RjyAg                                                          
Downloaded "Tove Lo - Habits (Stay High)": https://music.youtube.com/watch?v=hXisXwT44Gw                                                    
OSError: [Errno 24] Too many open files                                                                                                     
OSError: [Errno 24] Too many open files                                                                                                     
Couldn't read cache at: /home/lpryszcz/.spotdl/.spotipy
Couldn't read cache at: /home/lpryszcz/.spotdl/.spotipy
Couldn't read cache at: /home/lpryszcz/.spotdl/.spotipy


### Other details

_No response_