search

nathom / streamrip

A scriptable music downloader for Qobuz, Tidal, SoundCloud, and Deezer
GNU General Public License v3.0
2.66k stars 234 forks source link

[BUG] [Errno 24] Too many open files with SoundCloud #450

Closed ethanarns closed 11 months ago

ethanarns commented 1 year ago

Describe the bug

When attempted to download a set from SoundCloud, if there are more than 9 items, it often breaks for the remaining items

Command Used

rip url "https://soundcloud.com/aetherrealm-music/sets/redneck-vikings-from-hell"

Debug Traceback

xZcGmqSCcQw__&Key-Pair-Id=APKAI6TU7MMXM5DG6EPQ': '/var/folders/q_/b5fbcgh962jdptcmdt71wgp00000gn/T/__streamrip_partial_4461050543525239864', 'https://cf-hls-media.sndcdn.com/media/159660/4310412/4314590/gtOn1KILL0WE.128.mp3?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLWhscy1tZWRpYS5zbmRjZG4uY29tL21lZGlhLzE1OTY2MC8qLyovZ3RPbjFLSUxMMFdFLjEyOC5tcDMiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2OTMzMzM3OTJ9fX1dfQ__&Signature=LQf0tZlt3XJ3ZuiDCM3LjVR3rMO9Y1KThyznSdR3di~bhNmG4STLxhsOFDEH6W7BSK91PkjZtLQhg2AiSxrI6779RR101EYL29E3yC~-ZNXuvRQv4U2cSeA84RXsaLYhi8YBIuOOGQalm1uiq1SAB72keggsSbeJpNZVmZWcyE7ZUsTUYCbpnGc-10N-S6F5NFkmDZfx7f-87nJ3cuCgpeVWYm3rU2Dh0JgxQwsHlIZ1msSdyqcFwlI-3nNMDEAFF~UkpS3LT-lnYZtnDcQGwRm3W5QaqFPZXQmkjW-fR7BECXFp05vktTUnfZrCU9CckPud8jH6X33xZcGmqSCcQw__&Key-Pair-Id=APKAI6TU7MMXM5DG6EPQ': '/var/folders/q_/b5fbcgh962jdptcmdt71wgp00000gn/T/__streamrip_partial_3775013815221591000'}

Track 11 |██████████| 00:00 left at 25.7Chunk/s 
DEBUG:streamrip:Downloaded: /Users/username/StreamripDownloads/Woods 5 Grey Skies & Electric Light/11. Woods of Ypres - Alternate Ending.mp3 -> /Users/username/StreamripDownloads/Woods 5 Grey Skies & Electric Light/11. Woods of Ypres - Alternate Ending.mp3
DEBUG:streamrip:Tagging file with MP3 container
DEBUG:streamrip:Excluded tags: set()
DEBUG:streamrip:Setting TIT2 tag to Alternate Ending
DEBUG:streamrip:Setting TPE1 tag to Woods of Ypres
DEBUG:streamrip:Setting TALB tag to Woods 5: Grey Skies & Electric Light
DEBUG:streamrip:Setting TPE2 tag to Woods of Ypres
DEBUG:streamrip:Setting TYER tag to 2017
DEBUG:streamrip:Setting TCON tag to Rock
DEBUG:streamrip:Setting TCOP tag to 2011 2011 Earache Records Ltd
DEBUG:streamrip:Setting TRCK tag to 11/0
DEBUG:streamrip:Setting TPOS tag to 1/1

  Stack trace:

  4  /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:980 in _wrap_create_connection
       978│         try:
       979│             async with ceil_timeout(timeout.sock_connect):
    →  980│                 return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
       981│         except cert_errors as exc:
       982│             raise ClientConnectorCertificateError(req.connection_key, exc) from exc

  3  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py:1085 in create_connection
      1083│                 try:
      1084│                     if len(exceptions) == 1:
    → 1085│                         raise exceptions[0]
      1086│                     else:
      1087│                         # If they all have the same str(), raise one.

  2  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py:1069 in create_connection
      1067│                 for addrinfo in infos:
      1068│                     try:
    → 1069│                         sock = await self._connect_sock(
      1070│                             exceptions, addrinfo, laddr_infos)
      1071│                         break

  1  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py:950 in _connect_sock
       948│         sock = None
       949│         try:
    →  950│             sock = socket.socket(family=family, type=type_, proto=proto)
       951│             sock.setblocking(False)
       952│             if local_addr_infos is not None:

  OSError

  [Errno 24] Too many open files

  at /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py:232 in __init__
      228│             if type == -1:
      229│                 type = SOCK_STREAM
      230│             if proto == -1:
      231│                 proto = 0
    → 232│         _socket.socket.__init__(self, family, type, proto, fileno)
      233│         self._io_refs = 0
      234│         self._closed = False
      235│ 
      236│     def __enter__(self):

The following error occurred when trying to handle this error:

  Stack trace:

  27  /usr/local/lib/python3.11/site-packages/cleo/application.py:330 in run
       328│ 
       329│             try:
     → 330│                 exit_code = self._run(io)
       331│             except Exception as e:
       332│                 if not self._catch_exceptions:

  26  /usr/local/lib/python3.11/site-packages/rip/cli.py:762 in _run
       760│             logger.addHandler(fh)
       761│ 
     → 762│         super()._run(io)
       763│ 
       764│     def create_io(self, input=None, output=None, error_output=None):

  25  /usr/local/lib/python3.11/site-packages/cleo/application.py:425 in _run
       423│                 io.set_input(ArgvInput(argv))
       424│ 
     → 425│         exit_code = self._run_command(command, io)
       426│         self._running_command = None
       427│ 

  24  /usr/local/lib/python3.11/site-packages/cleo/application.py:432 in _run_command
       430│     def _run_command(self, command: Command, io: IO) -> int:
       431│         if self._event_dispatcher is None:
     → 432│             return command.run(io)
       433│ 
       434│         # Bind before the console.command event,

  23  /usr/local/lib/python3.11/site-packages/cleo/commands/base_command.py:118 in run
       116│         io.input.validate()
       117│ 
     → 118│         status_code = self.execute(io)
       119│ 
       120│         if status_code is None:

  22  /usr/local/lib/python3.11/site-packages/cleo/commands/command.py:85 in execute
        83│ 
        84│         try:
     →  85│             return self.handle()
        86│         except KeyboardInterrupt:
        87│             return 1

  21  /usr/local/lib/python3.11/site-packages/rip/cli.py:125 in handle
       123│ 
       124│         if len(core) > 0:
     → 125│             core.download()
       126│         elif not urls and path is None:
       127│             self.line("Must pass arguments. See rip url -h.")

  20  /usr/local/lib/python3.11/site-packages/rip/core.py:314 in download
       312│ 
       313│             try:
     → 314│                 item.download(**arguments)
       315│                 for item_id in item.downloaded_ids:
       316│                     # Add items row by row

  19  /usr/local/lib/python3.11/site-packages/streamrip/media.py:1175 in download
       1173│                     for future in future_map.keys():
       1174│                         try:
     → 1175│                             future.result()
       1176│                         except NonStreamable as e:
       1177│                             item = future_map[future]

  18  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py:449 in result
       447│                     raise CancelledError()
       448│                 elif self._state == FINISHED:
     → 449│                     return self.__get_result()
       450│ 
       451│                 self._condition.wait(timeout)

  17  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py:401 in __get_result
       399│         if self._exception:
       400│             try:
     → 401│                 raise self._exception
       402│             finally:
       403│                 # Break a reference cycle with the exception in self._exception

  16  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py:58 in run
        56│ 
        57│         try:
     →  58│             result = self.fn(*self.args, **self.kwargs)
        59│         except BaseException as exc:
        60│             self.future.set_exception(exc)

  15  /usr/local/lib/python3.11/site-packages/streamrip/media.py:1880 in _download_item
       1878│             item.meta.discnumber = 1
       1879│ 
     → 1880│         item.download(**kwargs)
       1881│ 
       1882│         item.tag(

  14  /usr/local/lib/python3.11/site-packages/streamrip/media.py:361 in download
        359│ 
        360│         elif self.client.source == "soundcloud":
     →  361│             self._soundcloud_download(dl_info)
        362│ 
        363│         else:

  13  /usr/local/lib/python3.11/site-packages/streamrip/media.py:450 in _soundcloud_download
        448│                     bar.update(1)
        449│ 
     →  450│                 pool.download(callback=update_tqdm_bar)
        451│ 
        452│                 concat_audio_files(pool.files, self.path, "mp3")

  12  /usr/local/lib/python3.11/site-packages/streamrip/downloadtools.py:198 in download
       196│     def download(self, callback=None):
       197│         self.callback = callback
     → 198│         asyncio.run(self._download_urls())
       199│ 
       200│     @property

  11  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py:190 in run
       188│ 
       189│     with Runner(debug=debug) as runner:
     → 190│         return runner.run(main)
       191│ 
       192│ 

  10  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py:118 in run
       116│         self._interrupt_count = 0
       117│         try:
     → 118│             return self._loop.run_until_complete(task)
       119│         except exceptions.CancelledError:
       120│             if self._interrupt_count > 0:

   9  /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py:653 in run_until_complete
        651│             raise RuntimeError('Event loop stopped before Future completed.')
        652│ 
     →  653│         return future.result()
        654│ 
        655│     def stop(self):

   8  /usr/local/lib/python3.11/site-packages/streamrip/downloadtools.py:181 in _download_urls
       179│                 for url in self.urls.values()
       180│             ]
     → 181│             await asyncio.gather(*tasks)
       182│ 
       183│     async def _download_url(self, session, url):

   7  /usr/local/lib/python3.11/site-packages/streamrip/downloadtools.py:186 in _download_url
       184│         filename = await self.getfn(url)
       185│         logger.debug("Downloading %s", url)
     → 186│         async with session.get(url) as response, aiofiles.open(filename, "wb") as f:
       187│             # without aiofiles     3.6632679780000004s
       188│             # with    aiofiles     2.504482839s

   6  /usr/local/lib/python3.11/site-packages/aiohttp/client.py:1141 in __aenter__
       1139│ 
       1140│     async def __aenter__(self) -> _RetType:
     → 1141│         self._resp = await self._coro
       1142│         return self._resp
       1143│ 

   5  /usr/local/lib/python3.11/site-packages/aiohttp/client.py:536 in _request
        534│                         async with ceil_timeout(real_timeout.connect):
        535│                             assert self._connector is not None
     →  536│                             conn = await self._connector.connect(
        537│                                 req, traces=traces, timeout=real_timeout
        538│                             )

   4  /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:540 in connect
        538│ 
        539│             try:
     →  540│                 proto = await self._create_connection(req, traces, timeout)
        541│                 if self._closed:
        542│                     proto.close()

   3  /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:901 in _create_connection
        899│             _, proto = await self._create_proxy_connection(req, traces, timeout)
        900│         else:
     →  901│             _, proto = await self._create_direct_connection(req, traces, timeout)
        902│ 
        903│         return proto

   2  /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:1209 in _create_direct_connection
       1207│         else:
       1208│             assert last_exc is not None
     → 1209│             raise last_exc
       1210│ 
       1211│     async def _create_proxy_connection(

   1  /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:1178 in _create_direct_connection
       1176│ 
       1177│             try:
     → 1178│                 transp, proto = await self._wrap_create_connection(
       1179│                     self._factory,
       1180│                     host,

  ClientConnectorError

  Cannot connect to host cf-hls-media.sndcdn.com:443 ssl:default [Too many open files]

  at /usr/local/lib/python3.11/site-packages/aiohttp/connector.py:988 in _wrap_create_connection
       984│             raise ClientConnectorSSLError(req.connection_key, exc) from exc
       985│         except OSError as exc:
       986│             if exc.errno is None and isinstance(exc, asyncio.TimeoutError):
       987│                 raise
    →  988│             raise client_error(req.connection_key, exc) from exc
       989│ 
       990│     def _fail_on_no_start_tls(self, req: "ClientRequest") -> None:
       991│         """Raise a :py:exc:`RuntimeError` on missing ``start_tls()``.
       992│ 

If this was unexpected, please open a Bug Report at https://github.com/nathom/streamrip/issues/new/choose

Track 09 |          | ? left at ?Chunk/s 

Config File

[downloads]
# Folder where tracks are downloaded to
folder = "/Users/username/StreamripDownloads"
# Put Qobuz albums in a 'Qobuz' folder, Tidal albums in 'Tidal' etc.
source_subdirectories = false

[downloads.concurrency]
# Download (and convert) tracks all at once, instead of sequentially. 
# If you are converting the tracks, or have fast internet, this will 
# substantially improve processing speed.
enabled = true
# The maximum number of tracks to download at once
# If you have very fast internet, you will benefit from a higher value,
# A value that is too high for your bandwidth may cause slowdowns
max_connections = 3

[qobuz]
# 1: 320kbps MP3, 2: 16/44.1, 3: 24/<=96, 4: 24/>=96
quality = 3
# This will download booklet pdfs that are included with some albums
download_booklets = true

email = ""
# This is an md5 hash of the plaintext password
password = ""
# Do not change
app_id = ""
# Do not change
secrets = []

[tidal]
# 0: 256kbps AAC, 1: 320kbps AAC, 2: 16/44.1 "HiFi" FLAC, 3: 24/44.1 "MQA" FLAC
quality = 3
# This will download videos included in Video Albums.
download_videos = true

# Do not change any of the fields below
user_id = ""
country_code = ""
access_token = ""
refresh_token = ""
# Tokens last 1 week after refresh. This is the Unix timestamp of the expiration
# time. If you haven't used streamrip in more than a week, you may have to log
# in again using `rip config --tidal`
token_expiry = ""

[deezer]
# 0, 1, or 2
# This only applies to paid Deezer subscriptions. Those using deezloader
# are automatically limited to quality = 1
quality = 2
# An authentication cookie that allows streamrip to use your Deezer account
# See https://github.com/nathom/streamrip/wiki/Finding-Your-Deezer-ARL-Cookie
# for instructions on how to find this
arl = ""
# This allows for free 320kbps MP3 downloads from Deezer
# If an arl is provided, deezloader is never used
use_deezloader = true
# This warns you when the paid deezer account is not logged in and rip falls
# back to deezloader, which is unreliable
deezloader_warnings = true

[soundcloud]
# Only 0 is available for now
quality = 0
# This changes periodically, so it needs to be updated
client_id = "z59xjnxSZIusnBJv9W3cAnV5rNDF9WpL"
app_version = "1692695220"

[youtube]
# Only 0 is available for now
quality = 0
# Download the video along with the audio
download_videos = false
# The path to download the videos to
video_downloads_folder = ""

# This stores a list of item IDs so that repeats are not downloaded.

[database.downloads]
enabled = true
path = "/Users/ethanarns/Library/Application Support/streamrip/downloads.db"

# If a download fails, the item ID is stored here. Then, `rip repair` can be
# called to retry the downloads

[database.failed_downloads]
enabled = true
path = "/Users/ethanarns/Library/Application Support/streamrip/failed_downloads.db"

# Convert tracks to a codec after downloading them.

[conversion]
enabled = false
# FLAC, ALAC, OPUS, MP3, VORBIS, or AAC
codec = "ALAC"
# In Hz. Tracks are downsampled if their sampling rate is greater than this. 
# Value of 48000 is recommended to maximize quality and minimize space
sampling_rate = 48000
# Only 16 and 24 are available. It is only applied when the bit depth is higher
# than this value.
bit_depth = 24
# Only applicable for lossy codecs
lossy_bitrate = 320

# Filter a Qobuz artist's discography. Set to 'true' to turn on a filter.

[filters]
# Remove Collectors Editions, live recordings, etc.
extras = false
# Picks the highest quality out of albums with identical titles.
repeats = false
# Remove EPs and Singles
non_albums = false
# Remove albums whose artist is not the one requested
features = false
# Skip non studio albums
non_studio_albums = false
# Only download remastered albums
non_remaster = false

[artwork]
# Write the image to the audio file
embed = true
# The size of the artwork to embed. Options: thumbnail, small, large, original.
# "original" images can be up to 30MB, and may fail embedding. 
# Using "large" is recommended.
size = "large"
# Both of these options limit the size of the embedded artwork. If their values
# are larger than the actual dimensions of the image, they will be ignored.
# If either value is -1, the image is left untouched.
max_width = -1
max_height = -1
# Save the cover image at the highest quality as a seperate jpg file
keep_hires_cover = true

[metadata]
# Sets the value of the 'ALBUM' field in the metadata to the playlist's name. 
# This is useful if your music library software organizes tracks based on album name.
set_playlist_to_album = true
# Replaces the original track's tracknumber with it's position in the playlist
new_playlist_tracknumbers = true
# The following metadata tags won't be applied
# See https://github.com/nathom/streamrip/wiki/Metadata-Tag-Names for more info
exclude = []

# Changes the folder and file names generated by streamrip.

[filepaths]
# Create folders for single tracks within the downloads directory using the folder_format
# template
add_singles_to_folder = false
# Available keys: "albumartist", "title", "year", "bit_depth", "sampling_rate",
# "container", "id", and "albumcomposer"
folder_format = "{albumartist} - {title} ({year}) [{container}] [{bit_depth}B-{sampling_rate}kHz]"
# Available keys: "tracknumber", "artist", "albumartist", "composer", "title",
# and "albumcomposer"
track_format = "{tracknumber}. {artist} - {title}{explicit}"
# Only allow printable ASCII characters in filenames.
restrict_characters = false
# Truncate the filename if it is greater than 120 characters
# Setting this to false may cause downloads to fail on some systems
truncate = true

# Last.fm playlists are downloaded by searching for the titles of the tracks

[lastfm]
# The source on which to search for the tracks.
source = "qobuz"
# If no results were found with the primary source, the item is searched for 
# on this one.
fallback_source = "deezer"

[theme]
# Options: "dainty" or "plain"
progress_bar = "dainty"

[misc]
# Metadata to identify this config file. Do not change.
version = "1.9.6"

Operating System

macOs

streamrip version

1.9.7

Screenshots and recordings

No response

Additional context

It also seems to rename the tracks in the wrong order, but I feel like that might be connected to this

Aubinebn commented 1 year ago

Hi, " Cannot connect to host cf-hls-media.sndcdn.com:443 ssl:default [Too many open files]", this error let me think that there are too many request wich cannot be handled.

Maybe you could try to change the download concurrency parameters on the config file. To disable download concurrency do change the value of enabled from true to false. (line 12)

Let me know if this work for you.

stale[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.