Download error: KeyError: 'playNavigationEndpoint'

itmax78 commented 3 years ago

In the evening I started downloading two playlists, they downloaded normally. In the morning I start downloading the next playlist and get an error:

Fetching Playlist...                                                                                                                                                                                                                  
Traceback (most recent call last):
File "/root/envspotdl/bin/spotdl", line 8, in <module>
sys.exit(console_entry_point())
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/__main__.py", line 114, in console_entry_point
 songObjList = get_playlist_tracks(request)
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
 song = SongObj.from_url(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/songObj.py", line 66, in from_url
  youtubeLink = SongObj.searchProvider(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 478, in search_and_get_best_match
 results = search_and_order_ytm_results(
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 366, in search_and_order_ytm_results
  results = __query_and_simplify(songSearchStr)
File "/root/envspotdl/lib/python3.8/site-packages/spotdl/search/provider.py", line 176, in __query_and_simplify
  linkBlock = contents['musicResponsiveListItemRenderer'] \
KeyError: 'playNavigationEndpoint'

I installed spotdl like this:

pip3 install spotdl
pip3 uninstall -y pytube3
pip3 install git+https://github.com/nficano/pytube

other installation options did not solve the problem. When installing from master branch, I get another error: TypeError: cannot unpack non-iterable NoneType object

issue-label-bot[bot] commented 3 years ago

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.98. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

itmax78 commented 3 years ago

A little more information, maybe it will be useful. Tried redirecting traffic through Tor to bypass YouTube blocking, it didn't work. I get similar errors on three different servers, on different subnets. I'm using Ubuntu 18.04 and CentOS 7

Silverarmor commented 3 years ago

2nd error (from master branch install) Appears to be duplicate of #1025 further down.

Try the following: Try the following:

pip install pip-autoremove
pip-autoremove spotdl
pip install https://codeload.github.com/spotDL/spotify-downloader/zip/master
pip uninstall pytube
pip install git+https://github.com/nficano/pytube
spotdl $trackUrl

itmax78 commented 3 years ago

Now I get both errors at once, a complete list of what I did:

> python3.8 -m venv master
> source master/bin/activate
> pip install pip-autoremove
> pip-autoremove spotdl
> pip install https://codeload.github.com/spotDL/spotify-downloader/zip/master
> pip uninstall pytube
> pip install git+https://github.com/nficano/pytube
> spotdl https://open.spotify.com/playlist/3lsjGhv5VTN0mjnmDP3HI9?si=1Y6rH7qCRJq8k1H7RSpJOw
> 
> Fetching Playlist...                                                                                                                                                                                                                  
> Traceback (most recent call last):
>   File "/root/master/bin/spotdl", line 33, in <module>
>     sys.exit(load_entry_point('spotdl==3.1.4', 'console_scripts', 'spotdl')())
>   File "/root/master/lib/python3.8/site-packages/spotdl/__main__.py", line 111, in console_entry_point
>     songObjList = get_playlist_tracks(request)
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
>     song = SongObj.from_url(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/songObj.py", line 66, in from_url
>     youtubeLink = SongObj.searchProvider(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 478, in search_and_get_best_match
>     results = search_and_order_ytm_results(
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 366, in search_and_order_ytm_results
>     results = __query_and_simplify(songSearchStr)
>   File "/root/master/lib/python3.8/site-packages/spotdl/search/provider.py", line 176, in __query_and_simplify
>     linkBlock = contents['musicResponsiveListItemRenderer'] \
> KeyError: 'playNavigationEndpoint'
> Exception ignored in: <function tqdm.__del__ at 0x7f502d906820>
> Traceback (most recent call last):
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1124, in __del__
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1337, in close
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1516, in display
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1127, in __repr__
>   File "/root/master/lib/python3.8/site-packages/spotdl/download/progressHandlers.py", line 23, in format_dict
>   File "/root/master/lib/python3.8/site-packages/tqdm/std.py", line 1477, in format_dict
> TypeError: cannot unpack non-iterable NoneType object

tytydraco commented 3 years ago

I'm getting this error as well. @Silverarmor's solution did not help, sadly.

KeyError: 'playNavigationEndpoint'

s1as3r commented 3 years ago

While Debugging, here's what I found:

When searching for a song, spotdl gathers all the results that the youtube music api provides.
If one of the songs in the results is not available in a country, 'playNavigationEndpoint' key would not be present in it's result, hence this error.
In this specific case, while searching for the song 'KUURO - Don't Stop', youtube music api also returns a result for a song 'KUURO - Rapture' which isn't available in some countries and it causes the error.
- If you try to download the song 'KUURO - Don't Stop' you'd get the same error.
- Downloading any other song won't give this error.

The Fix:

I will be opening a PR, till it gets merged, here's how you can fix it manually.

Add a try, except block in site-packages/spotdl/search/provider.py line 176 like this:

try:
linkBlock = contents['musicResponsiveListItemRenderer'] \
['overlay'] \
    ['musicItemThumbnailOverlayRenderer'] \
        ['content'] \
            ['musicPlayButtonRenderer'] \
                ['playNavigationEndpoint']
except KeyError:
continue

You can also try and remove "KUURO - Don't Stop" from this playlist and try again.

Please close this issue, if this fixes it.

tytydraco commented 3 years ago

Please close this issue, if this fixes it.

It could be a good idea to put an error message there too; not sure how the rest of the program handles errors. Also, it could be a bad idea to wrap that whole thing. If there's an error in the future regarding any of those keys, we would never be able to know. Maybe we can do something like this:

_linkBlock = contents['musicResponsiveListItemRenderer'] \
        ['overlay'] \
            ['musicItemThumbnailOverlayRenderer'] \
                ['content'] \
                    ['musicPlayButtonRenderer']
try:
    linkBlock = _linkBlock ['playNavigationEndpoint']
except KeyError:
    continue

s1as3r commented 3 years ago

Also, it could be a bad idea to wrap that whole thing

Yup, You're right. Will make the suggested changes.

lvlzyro commented 3 years ago

Here is what I got when I add that block:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3.8/site-packages/urllib3/util/retry.py", line 403, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.spotify.com', port=443): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 114, in console_entry_point
    songObjList = get_playlist_tracks(request)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/search/utils.py", line 82, in get_playlist_tracks
    song = SongObj.from_url(
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/search/songObj.py", line 41, in from_url
    rawTrackMeta = spotifyClient.track(spotifyURL)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 338, in track
    return self._get("tracks/" + trid)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 291, in _get
    return self._internal_call("GET", url, payload, kwargs)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotipy/client.py", line 240, in _internal_call
    response = self._session.request(
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.spotify.com', port=443): Read timed out. (read timeout=5)

tzarad commented 3 years ago

For me all youtubelink are None

itmax78 commented 3 years ago

Corrected the code as suggested above. When downloading any individual songs, I get:

spotdl https://open.spotify.com/track/2iyVwlrSE1jJOrohg5tv0z?si=jPNPRwi6T3W-LKkBq6e3PA Fetching Song...
Skipping Don't Stop (https://open.spotify.com/track/2iyVwlrSE1jJOrohg5tv0z?si=jPNPRwi6T3W-LKkBq6e3PA) as no match could be found on youtube 0%| |ETA: ?, ~min/song

I launch a small playlist:

spotdl https://open.spotify.com/playlist/37i9dQZF1DWVi45nh2EuPP?si=phjVN11FSCOaQNrwXQ4pLg Fetching Playlist...
Traceback (most recent call last): File "/root/master/bin/spotdl", line 8, in sys.exit(console_entry_point()) File "/root/master/lib/python3.8/site-packages/spotdl/main.py", line 116, in console_entry_point downloader.download_multiple_songs(songObjList) File "/root/master/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs self.downloadTracker.load_song_list(songObjList) File "", line 2, in load_song_list File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod raise convert_to_error(kind, result) TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

I launch the playlist from my first message:

https://open.spotify.com/playlist/3lsjGhv5VTN0mjnmDP3HI9?si=1Y6rH7qCRJq8k1H7RSpJOw Fetching Playlist... and so on for more than two hours :(

Strangely enough, I made another virtual environment, fixed only the code and launched the playlist in it that I managed to download for the first time, now the process is also frozen.

s1as3r commented 3 years ago

For me all youtubelink are None

Corrected the code as suggested above. When downloading any individual songs, I get:

That's a different issue altogether. See #1039

s1as3r commented 3 years ago

Here is what I got when I add that block:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

.....

This seems to be an issue with requests. Retry downloading a few times.

lvlzyro commented 3 years ago

This seems to be an issue with requests. Retry downloading a few times.

Here is what I get which is same with itmax78's one:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 116, in console_entry_point
    downloader.download_multiple_songs(songObjList)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs
    self.downloadTracker.load_song_list(songObjList)
  File "<string>", line 2, in load_song_list
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

s1as3r commented 3 years ago

This seems to be an issue with requests. Retry downloading a few times.

Here is what I get which is same with itmax78's one:

Fetching Playlist...                                                           
Traceback (most recent call last):
  File "/home/lvlzyro/.local/bin/spotdl", line 8, in <module>
    sys.exit(console_entry_point())
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/__main__.py", line 116, in console_entry_point
    downloader.download_multiple_songs(songObjList)
  File "/home/lvlzyro/.local/lib/python3.8/site-packages/spotdl/download/downloader.py", line 288, in download_multiple_songs
    self.downloadTracker.load_song_list(songObjList)
  File "<string>", line 2, in load_song_list
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
TypeError: remove: path should be string, bytes or os.PathLike, not NoneType

See #968 and #929

lvlzyro commented 3 years ago

See #968 and #929

YouTube Music is available in my country and I tried to install and uninstall pytube but it did not help.

kommersch commented 3 years ago

I have the same issue as lvlzyro. And yes, Youtube Music is available in my country as well.

ghost commented 3 years ago

A fix for this is in the pipeline. Please wait a few days and install from master.

Neltherion commented 3 years ago

This issue hasn't been fixed yet. I'm getting:

Fetching Song...
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\spotdl.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\__main__.py", line 94, in console_entry_point
    song = SongObj.from_url(request)
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\songObj.py", line 70, in from_url
    duration
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 480, in search_and_get_best_match
    songAlbumName, songDuration
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 366, in search_and_order_ytm_results
    results = __query_and_simplify(songSearchStr)
  File "c:\programdata\anaconda3\lib\site-packages\spotdl\search\provider.py", line 181, in __query_and_simplify
    ['playNavigationEndpoint']
KeyError: 'playNavigationEndpoint'
  0%|          |ETA: ?, ~min/song

YouTube Music is available along YouTube.

EDIT: This made it work again.

spotDL / spotify-downloader

Download error: KeyError: 'playNavigationEndpoint' #1038