search

justin025 / onthespot

qt based music downloader written in python
GNU General Public License v2.0
10 stars 0 forks source link

Getting a lot of failed downloads #3

Open RandomLegend opened 1 week ago

RandomLegend commented 1 week ago

Putting in a playlist of mine i get a lot of failed downloads, i'd say roughly 50% of the playlist is failing, the songs do definetely exist on spotify and are playable.

This is the terminal output for on of them, they all look like this:

[2024-09-02 23:27:53,707 :: spotutils :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/utils/spotify.py -> 103:       sanitize_data() :: INFO] -> Sanitising string: "Stellar"; Allow path separators: False
[2024-09-02 23:27:53,707 :: spotutils :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/utils/spotify.py -> 103:       sanitize_data() :: INFO] -> Sanitising string: "Masquerade"; Allow path separators: False
[2024-09-02 23:27:53,707 :: spotutils :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/utils/spotify.py -> 103:       sanitize_data() :: INFO] -> Sanitising string: "Masquerade"; Allow path separators: False
[2024-09-02 23:27:53,707 :: spotutils :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/utils/spotify.py -> 103:       sanitize_data() :: INFO] -> Sanitising string: "Stellar"; Allow path separators: False
[2024-09-02 23:27:53,707 :: spotutils :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/utils/spotify.py -> 103:       sanitize_data() :: INFO] -> Sanitising string: "2022 Stellar"; Allow path separators: False
Audio key error, code: 2
Audio key error, code: 2
[2024-09-02 23:27:54,235 :: worker.downloader.SESSION_DL_TH-768b3b82-2ebe-43d8-88a8-370351388c98 :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/worker/downloader.py -> 221:      download_track() :: ERROR] -> Download failed for track by id '6UnOxNp0jZfFXyKcAtJwUj', Unexpected error: Traceback (most recent call last):
  File "/home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/worker/downloader.py", line 122, in download_track
    stream = session.content_feeder().load(track_id, VorbisOnlyAudioQuality(quality), False, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 739, in load
    return self.load_track(playable_id, audio_quality_picker, preload,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 800, in load_track
    return self.load_stream(file, track, None, preload, halt_listener)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 754, in load_stream
    return CdnFeedHelper.load_track(self.__session, track, file,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 339, in load_track
    key = session.audio_key().get_audio_key(track.gid, file.file_id)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 277, in get_audio_key
    return self.get_audio_key(gid, file_id, False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/RandomLegend/Downloads/onthespot/onthespot/venv/lib/python3.12/site-packages/librespot/audio/__init__.py", line 278, in get_audio_key
    raise RuntimeError(
RuntimeError: Failed fetching audio key! gid: (dontknowifthisissecret), fileId: (dontknowifthisissecret)
 !
[2024-09-02 23:27:54,235 :: worker.downloader.SESSION_DL_TH-768b3b82-2ebe-43d8-88a8-370351388c98 :: /home/RandomLegend/Downloads/onthespot/onthespot/src/onthespot/worker/downloader.py -> 384:                 run() :: ERROR] -> Download process returned false: 6UnOxNp0jZfFXyKcAtJwUj

i'm on arch linux, newest kernel, using the portable python version and installed the requirements inside a python venv.

RandomLegend commented 1 week ago

Alright, either it was completely restarting my PC (didn't think that could solve any issues with that :D) or it was the fact that i readded the playlist completely - either and / or both of those steps fixed the issue.

RandomLegend commented 1 week ago

Ha nope, just started doing it again!

After processing 66 songs, it started to fail again

RandomLegend commented 1 week ago

So after searching for that specific error i stumbled accross multiple github entries from various librespot implementations (1, 2, 3) and none of them seemed fixed 🤔

Letting it process the list until it's done, quitting the application and restarting the playlist seems to get me through it pass by pass. So i'd guess it'll take me around 7-10 restarts of the application to get my whole list downloaded.

I take what i can get, but maybe this is fixable?

justin025 commented 1 week ago

Sometimes when it fails on a specific song, in your case 67, it means there is something wrong with that song. From the people I've spoke to it could be a podcast duplicate or special characters in the title/metadata. You can try removing that song from your playlist, bulk downloading and then downloading that song separately.

I hear the original dev is working on a fix for the playlist api but it will take some time to be pushed upstream.

RandomLegend commented 1 week ago

Alright, thanks for that.

Would it be possible for you to make a button that would allow users to copy the links of all failed songs? So in my example i would paste my playlist link, let it go through, start to fail at some point and then there would be a button and with that button i'd have a list of all failed songs that i could paste again so it doesn't have to go through the playlist again?

maguirreg commented 1 week ago

The issue occurs with both playlists and podcasts. I traced it down to the call that retrieves information about the media item here. The problem isn't due to an error with the media itself; instead, I suspect it might be related to the interface with the source. It's possible they’ve implemented a request limit within a certain time frame. Even after increasing the download delay to 10, the error persists. Running with download delay of 15 gave me better results for a moment allowing me to download more items, but started to fail again after a couple more. I believe one solution would be to save all failed ones to retry them after, or implement something like a database with the MediaID and details to avoid making too many requests for the same unchanged data.

RandomLegend commented 1 week ago

Yeah after looking into my failed downloads i saw that there are some tracks in there that don't have any special character or anything.

Examaple - Runaway from The Corrs

Decreasing the amount of workers to "1" and increasing the wait time to "15" made it much more stable for me. I was able to download a whole playlist without a single fail now.

justin025 commented 1 week ago

The function you linked above seems to be for downloading episodes which are a little different, download_track is at lines 33-222 in the same file. It could definitely be a rate limit error but we would probably see a '400' http request somewhere in the log, see 204 for example. I do know a 'request cache' already exists but the problem is not metadata and songs in a playlist but the ogg file itself. If you guys are right its more likely that spotify has placed limits on the most bandwidth intensive media, that being streams.

I'll need to look into errors like these at some point. Thank you for sharing.

justin025 commented 1 week ago

I know some people are sensitive about sharing their playlists but would you mind sharing links to any playlists that break so I can compile a list to test. If you don't feel comfortable sharing publicly you can dm me on discord j075046

maguirreg commented 1 week ago

This is my last attempt:

  1. Download delay = 5, Max download Workers = 3
  2. Tried to download Case 63, for the first time after 5 hours without using the app.
  3. First run, it downloaded all 22 tracks. Cleared the completed/cancelled items and tried again
  4. Second run, failed on the 3rd, 7th, 14th, 15th, 17th-22nd tracks, same "Failed fetching audio key" error It suppose to skip the already downloaded items, but when is trying to get the info from the media it fails. image image output.log I'm attaching the logs for another execution where several others failed for the same list
justin025 commented 1 week ago

Definitely a rate limit issue. I downloaded the podcast and the first time it worked flawlessly. The second time it downloaded like so: 2024-09-03_10-37 I'm afraid I have no solution to this at the moment so I will reopen the issue until I can either add logging informing the user they are being rate limited or figure out a solution.

For any users experiencing this, just try closing and opening the app or waiting a little while and try again.

zerog13 commented 2 days ago

Changing the Download Delay to 25 has fixed my download rates 100%, even the metadata errors.

zerog13 commented 2 days ago

This is my last attempt:

1. Download delay = 5, Max download Workers = 3

2. Tried to download [Case 63](https://open.spotify.com/show/4c9ZKaFtEKweSYOlYvxfvp), for the first time after 5 hours without using the app.

3. First run, it downloaded all 22 tracks. Cleared the completed/cancelled items and tried again

4. Second run, failed on the 3rd, 7th, 14th, 15th, 17th-22nd tracks, same "Failed fetching audio key" error
   It suppose to skip the already downloaded items, but when is trying to get the info from the media it fails.
   ![image](https://private-user-images.githubusercontent.com/734215/363984409-bba9ceed-a833-4301-a07b-52112a78e6bf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYyMzM3NDUsIm5iZiI6MTcyNjIzMzQ0NSwicGF0aCI6Ii83MzQyMTUvMzYzOTg0NDA5LWJiYTljZWVkLWE4MzMtNDMwMS1hMDdiLTUyMTEyYTc4ZTZiZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTEzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxM1QxMzE3MjVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1mMTQ0YWMwZTg2OWFlMzU3MmQxMDk4ZjhlMDU4ZWQ1ZTBlYjhlMTRjYmFkMzllYzllZWIwMjUyZDhmNTIyZGJjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.DYz5mNU49eb6ZH1N691Or8MkregCHu9M9NnOjrAddDo)
   ![image](https://private-user-images.githubusercontent.com/734215/363988210-2fb95f51-c2d8-40e2-a8ac-28fafa276688.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYyMzM3NDUsIm5iZiI6MTcyNjIzMzQ0NSwicGF0aCI6Ii83MzQyMTUvMzYzOTg4MjEwLTJmYjk1ZjUxLWMyZDgtNDBlMi1hOGFjLTI4ZmFmYTI3NjY4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTEzJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxM1QxMzE3MjVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yZjNmYTEwMzNlZjgxMGIyZjFkMTZkZTVmZWY4NGUyMzQ5MmFkOTcxOGJmYmRlYmRlMGEzYjg0NjFiNThmODdkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.DmOLKiDPIm7eVHGVoSMU_X4g7aO7S30PZsB-skWmuQE)
   [output.log](https://github.com/user-attachments/files/16848974/output.log)
   I'm attaching the logs for another execution where several others failed for the same list

Change download rate to 25, it fixed my issue 100%. no errors on the last 350 tracks downloaded.

zerog13 commented 1 day ago

This is my last attempt:

1. Download delay = 5, Max download Workers = 3

2. Tried to download [Case 63](https://open.spotify.com/show/4c9ZKaFtEKweSYOlYvxfvp), for the first time after 5 hours without using the app.

3. First run, it downloaded all 22 tracks. Cleared the completed/cancelled items and tried again

4. Second run, failed on the 3rd, 7th, 14th, 15th, 17th-22nd tracks, same "Failed fetching audio key" error
   It suppose to skip the already downloaded items, but when is trying to get the info from the media it fails.
   ![image](https://private-user-images.githubusercontent.com/734215/363984409-bba9ceed-a833-4301-a07b-52112a78e6bf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYzMDE5NTEsIm5iZiI6MTcyNjMwMTY1MSwicGF0aCI6Ii83MzQyMTUvMzYzOTg0NDA5LWJiYTljZWVkLWE4MzMtNDMwMS1hMDdiLTUyMTEyYTc4ZTZiZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxNFQwODE0MTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1mOTkyYTczZThmNTBhYjljMzJjNjNlNzlkODE0ZjBlNmVkZDAzOThjN2RhN2VkZTA2NWU0NjY3YzliNTg2NTVlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.wqNRGXCHCztU35kWLOerkUNi0-jf5iHlWFR7v45UHEI)
   ![image](https://private-user-images.githubusercontent.com/734215/363988210-2fb95f51-c2d8-40e2-a8ac-28fafa276688.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYzMDE5NTEsIm5iZiI6MTcyNjMwMTY1MSwicGF0aCI6Ii83MzQyMTUvMzYzOTg4MjEwLTJmYjk1ZjUxLWMyZDgtNDBlMi1hOGFjLTI4ZmFmYTI3NjY4OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTE0JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkxNFQwODE0MTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lZWU2YmE4ODFhMDIyZDliOGQ3NmU1ODcwMTA2NDI3YzE4N2E2NDNlMGRlOGRkYzAxMjRiZWIxNTFiYTM2OTdmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.SmTogjukA8sy4iPxkHn6bJiD0KVK5T-IOZMcoWZ7DpE)
   [output.log](https://github.com/user-attachments/files/16848974/output.log)
   I'm attaching the logs for another execution where several others failed for the same list

try this plugin for podcasts - Podcasts - A player, downloader, transcriber https://podcasts.bluepill.life/index.html