Closed kosimst closed 2 years ago
Where about on this planet are you located? I've received a few similar reports from users in the past few days.
Just in case you are gathering stats/data: Same issue for me (in Denmark).
Same here (via Spotty integration in Logitech Media Server) (also Denmark)
@jesperll - you're not JesperDB who's been reporting this similar issue to me, are you?
That then would be three Danish users. It's hard to believe this was a coincidence...
I am from Austria @michaelherger, so maybe the problem is in europe.
Also saw similar channel errors from DK..
@jesperll - you're not JesperDB who's been reporting this similar issue to me, are you?
No, that's me and I'm also in Denmark :)
I have the same issue here: https://forums.slimdevices.com/showthread.php?111923-Announce-Spotty-4-0-integrate-local-library-with-your-Spotify-collection-(LMS-8-)&p=1049289&viewfull=1#post1049289
FWIW: A similar issue seem to happen if I try using new-api
branch, except the spotify client/UI on my PC/phone does not repeatedly skip fwd after ~5 secs [still no playback, though]
Console log with -v
from new-api
branch:
[...]
[2022-03-03T10:43:35Z TRACE librespot_connect::spirc] Frame has 48 tracks
[2022-03-03T10:43:35Z DEBUG librespot_playback::player] command=SetAutoNormaliseAsAlbum(false)
[2022-03-03T10:43:35Z DEBUG librespot_playback::player] command=Load(SpotifyId("spotify:track:2C0iEdLXanscPYGpwO2oSm"), true, 115962)
[2022-03-03T10:43:35Z TRACE librespot_connect::spirc] Sending status to server: [kPlayStatusPlay]
[2022-03-03T10:43:35Z DEBUG librespot::component] new SpClient
[2022-03-03T10:43:35Z INFO librespot_core::spclient] Resolved "gew1-spclient.spotify.com:443" as spclient access point
[2022-03-03T10:43:35Z DEBUG librespot::component] new TokenProvider
[2022-03-03T10:43:35Z TRACE librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2022-03-03T10:43:35Z TRACE librespot_connect::spirc] ==> kPlayStatusLoading
[2022-03-03T10:43:35Z TRACE librespot_connect::spirc] Sending status to server: [kPlayStatusLoading]
[2022-03-03T10:43:35Z ERROR librespot_core::mercury] error 403 for uri hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae
[2022-03-03T10:43:35Z ERROR librespot_playback::player] Unable to load audio item: Error { kind: Unavailable, error: Response(MercuryResponse { uri: "hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae", status_code: 403, payload: [[123, 34, 99, 111, 100, 101, 34, 58, 52, 44, 34, 101, 114, 114, 111, 114, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 34, 58, 34, 73, 110, 118, 97, 108, 105, 100, 32, 114, 101, 113, 117, 101, 115, 116, 34, 125]] }) }
[2022-03-03T10:43:35Z ERROR librespot_playback::player] Skipping to next track, unable to load track <SpotifyId("spotify:track:2C0iEdLXanscPYGpwO2oSm")>: ()
[2022-03-03T10:43:35Z ERROR librespot_core::session] could not dispatch command: Service unavailable { error handling Mercury response: MercuryResponse { uri: "hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae", status_code: 403, payload: [[123, 34, 99, 111, 100, 101, 34, 58, 52, 44, 34, 101, 114, 114, 111, 114, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 34, 58, 34, 73, 110, 118, 97, 108, 105, 100, 32, 114, 101, 113, 117, 101, 115, 116, 34, 125]] } }
[2022-03-03T10:43:35Z DEBUG librespot_connect::spirc] Marked <gid: "U\332\267\0046uOl\202YV\344\261\037\241\332" context: "NonPlayable"> at 1 as NonPlayable
[2022-03-03T10:43:35Z DEBUG librespot_playback::player] command=Preload(SpotifyId("spotify:track:40NRy5xeT32HCLae48DVov"))
[2022-03-03T10:43:35Z DEBUG librespot_playback::player] Preloading track
[2022-03-03T10:43:35Z TRACE librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2022-03-03T10:43:36Z ERROR librespot_core::mercury] error 403 for uri hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae
[2022-03-03T10:43:36Z ERROR librespot_core::session] could not dispatch command: Service unavailable { error handling Mercury response: MercuryResponse { uri: "hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae", status_code: 403, payload: [[123, 34, 99, 111, 100, 101, 34, 58, 52, 44, 34, 101, 114, 114, 111, 114, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 34, 58, 34, 73, 110, 118, 97, 108, 105, 100, 32, 114, 101, 113, 117, 101, 115, 116, 34, 125]] } }
[2022-03-03T10:43:36Z ERROR librespot_playback::player] Unable to load audio item: Error { kind: Unavailable, error: Response(MercuryResponse { uri: "hm://keymaster/token/authenticated?scope=playlist-read&client_id=65b708073fc0480ea92a077233ca87bd&device_id=ce8d71004f9597141d4b5940bd1bb2dc52a35dae", status_code: 403, payload: [[123, 34, 99, 111, 100, 101, 34, 58, 52, 44, 34, 101, 114, 114, 111, 114, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 34, 58, 34, 73, 110, 118, 97, 108, 105, 100, 32, 114, 101, 113, 117, 101, 115, 116, 34, 125]] }) }
[2022-03-03T10:43:36Z DEBUG librespot_playback::player] Unable to preload SpotifyId("spotify:track:40NRy5xeT32HCLae48DVov")
[...]
However, if I start librespot with -u <username> -p <password>
with same build from new-api
branch, I am able to play back the audio.
Maybe there were some recent changes to how the authentication is done with zeroconf?
Put simply, we can't do anything about channel errors. They sometimes hit the Spotify infrastructure and then go away after some time.
The panic is due to trying to reload a track that couldn't be loaded the first time. This is fixed in new-api
meaning that it won't panic anymore, but still won't be able to play the track.
There is no difference in authentication to the Spotify infrastructure between zeroconf and command line, all that matters is the way it receives the credentials (either from the command line or over the LAN). From there it authenticates against Spotify in the same way.
It's more likely that the time you logged in with command line credentials, you connected to an access point that didn't suffer from channel errors. You can check the access point in your (verbose) logs when librespot first connects.
I'm closing this as this is not something we can do anything about.
I do agree in general. But we could in theory provide an API to control what access point is used rather than always taking the first (or last, whatever it is, I find the comment there confusing). I am not saying that should then also be exposed to librespot application users (yet another program option) but it could.
Just sharing info, in case it's helpful: In DK. 2 official spotify clients working (1 not updated macOS, 1 latest releas iOS). They play just fine, except for trying to use Spotify Connect, with anything derived from or using librespot to enable Spotify Connect (E.g. Volumio 3 and spotifyd). Problem: Causes all tracks to skip.
i am seeing same thing here. also in denmark. skips all tracks in playlist - happend on 2 pi's at the same time. but because there where running a older version of dietpi i upgrade to bulseye and installed from scratch. with no luck.
All tracks get skipped and from journalctl -f i can see the error.
Would everybody mind sharing
Though there have been reports from Germany and Austria, too, I strongly believe this is either a geographical thing, or some network having issues or blocking things.
Of course! :) Here you go:
ISP: Hiper A/S (Bought by TDC, so they probably run on their network now) http://apresolve.spotify.com/ gives:
{"ap_list":["ap-gew4.spotify.com:4070","ap-gew4.spotify.com:443","ap-gew4.spotify.com:80","ap-guc3.spotify.com:4070","ap-gae2.spotify.com:443","ap-gew1.spotify.com:80"]}
I just did the test again and the client still does the skipping.
You will also want to lookup those host names and report their IP. Same names could result in different IPs depending on geography, I’m not sure.
ap-gew4.spotify.com: 34.158.0.131 ap-guc3.spotify.com: 104.154.127.126 ap-gae2.spotify.com: 104.199.240.237 ap-gew1.spotify.com: 104.199.65.124
From The Netherlands (Ziggo) I get the same IP addresses for those hosts. My apresolve
answer looks like this and I can play tracks fine:
{"ap_list":["ap-gew1.spotify.com:4070","ap-gew1.spotify.com:443","ap-gew1.spotify.com:80","ap-gew1.spotify.com:4070","ap-guc3.spotify.com:443","ap-gae2.spotify.com:80"]}
librespot
picks the first access point from the list, in my case ap-gew1.spotify.com:4070
. Again you can see this in your verbose logs.
Interesting, mine chooses the ap-gew4:
[2022-03-04T11:10:10Z INFO librespot_core::session] Connecting to AP "ap-gew4.spotify.com:4070"
And then starts skipping with these in the log:
[2022-03-04T11:10:12Z DEBUG librespot_connect::spirc] At track 4 of 30 <"spotify:playlist:37i9dQZEVXcEqdyS2qRIO4"> update [false]
[2022-03-04T11:10:12Z TRACE librespot_connect::spirc] Sending status to server: [kPlayStatusPlay]
[2022-03-04T11:10:12Z DEBUG librespot_playback::player] command=Load(SpotifyId { id: 239161766187612460604504651615889269027, audio_type: Track }, true, 0)
[2022-03-04T11:10:12Z TRACE librespot_connect::spirc] Sending status to server: [kPlayStatusPlay]
[2022-03-04T11:10:12Z INFO librespot_playback::player] Loading <Uku> with Spotify URI <spotify:track:5tvSDMa9x46Rdd4ndtJBBh>
[2022-03-04T11:10:12Z DEBUG librespot_audio::fetch] Downloading file 3f9ee5484ef61d6d72cfccd30446eb01e2dee49a
[2022-03-04T11:10:12Z ERROR librespot_core::channel] channel error: 2 1
[2022-03-04T11:10:12Z ERROR librespot_playback::player] Unable to load encrypted file: ChannelError
[2022-03-04T11:10:12Z WARN librespot_playback::player] Unable to load <SpotifyId { id: 239161766187612460604504651615889269027, audio_type: Track }>
Skipping to next track
I just tried to force the ap-gew4.spotify.com
domain to point to the IP of ap-gew1.spotify.com, by setting this in /etc/hosts:
104.199.65.124 ap-gew4.spotify.com
That works!! Sooo.... Is Spotify having an issue? Or are Spotify changing something?
Ha, you beat me to it! I wanted to test the opposite (using ap-gew4
instead of what I receive).
Yes, I believe there's a problem with that one host.
That's annoying... Have you experienced this before? In that case: how long does this normally last? I'd rather not do this trick on my music streamer, but if it's normally a problem that takes a long time to fix, I might need to :)
Thanks for your time guys! And thanks for this project! I love to have easy ability to get clean 320 bit directly into my DAC board :)
I don't think it's a common problem. But it's probably still worth trying to report to Spotify?
And should librespot be smarter and not only rely on the first element in the list?
I was thinking the same thing. Maybe there's some indicator that can be used to drop a host and try the other. I guess in this case it could be difficult, since the server and API is responding on some calls.
Channel errors are not new. If you use GitHub search on this project you will find a number of reports them. How long they'll last, only Spotify could say.
Programmatically detecting channel errors is not difficult, you literally see channel error: 2 1
in the log so we could act on that. But architecturally it's more difficult to let librespot
reconnect to another access point. In new-api
it's a bit easier to implement but still needs work.
Nice that there is something to detect it by :) But yeah... I can imagine how many little details has to be thought through in order for a total re-connect to work seamlessly. I wish I had learned Rust by now, so that I could try to help out.
i know it is not a beautiful solution but i can confirm that i works. as morphar suggested. :) thanks for the tip.
But instead of doing in all the pi's /etc/hosts we have implemented the redirect in our dns - 104.199.65.124 ponting to ap-gew4.spotify.com
@roderickvd Would you accept a PR with a "fix" where the host is picked at random instead of picking the first one returned? If the bad host is picked uniformly at random, natural trouble-shooting behaviour like restarting librespot would likely fix the issue. For example, if one out of five hosts are bad, the probability of picking the bad host twice in a row is just 4% (or viewed differently, there is a 20% chance the restart does not fix the issue). Though, I should say, apresolve.spotify.com returns three out of five bad hosts for me (same host on different ports), so it may be less effective in practice...
My reading of the host names is that the other two would be US Central and Asia East (Taiwan). Such choices could come at a cost in latency.
In Switzerland I received another set of European hosts. I therefore don’t think randomly picking one would be a good choice most of the time. I’d rather go with an option to force some host blame instead. But given my experience of the past 4-5 years would say it’s not worth the effort, as I can’t remember a similar incident.
They do seem like regional names; I agree that it's not a desirable solution then.
{"ap_list":["ap-gew4.spotify.com:4070","ap-gew4.spotify.com:443","ap-gew4.spotify.com:80","ap-gae2.spotify.com:4070","ap-gew1.spotify.com:443","ap-guc3.spotify.com:80"]}
Denmark here, and not working.
See @morphar's comment further up this report:
I just tried to force the
ap-gew4.spotify.com
domain to point to the IP of ap-gew1.spotify.com, by setting this in /etc/hosts:104.199.65.124 ap-gew4.spotify.com
That works!! Sooo.... Is Spotify having an issue? Or are Spotify changing something?
Hello, I just want to say thank you for fixing my Spotify Connent issues on Volumio.
@kingosticks's suggestion to offer an ABI to manually set the access point isn't a bad idea in this particular case. I am however hesitant to offer it as a command-line option because I don't want to get it misused.
What you can also try is blocking apresolve.spotify.com
altogether, which will cause librespot
to default to pick ap.spotify.com
as a fallback access point. I assume that this is geobalanced and might offer some other, in working order, access point.
I consider all of these as band-aids, ideally librespot
would reconnect and drop the erroneous access point. Like I said I can investigate this as part of new-api
. It's not on top of my list -- I'd rather have Spotify to fix their infrastructure -- but I should get around to it sometime. In the meantime PR's are more than welcome of course.
One way to enforce this behavior actually already is built in: set ap-port
to some random, invalid port. If librespot fails to connect to that port, it would use the fallback.
Ha I never thought about that, talk about emergent behaviour 😆
I didn't know about ap-port
until I started looking into how an override could be implemented 😄.
I got reports that things started working again. Can anybody confirm? Remove workaround and try again?
Doesn't seem to be working for me.
It doesn't work for me. Maybe the reports are false positives? Maybe they can stream, but it's because they're not connecting to ap-gew4.spotify.com for some reason?
Yes. It was a misunderstanding. The person was already using a workaround I implemented in my Spotty plugin (forcing fallback by providing an invalid ap-port
value). That seems to be working for people with the failing ap-gew4.spotify.com
.
Confirming that I also had this skipping issue here in Denmark for at least a week before finding the time to search for solutions.
Apresolve points to ap-gew4.spotify.com
- but through a VPN to Sweden/Stockholm I get ap-gew1.spotify.com
so this could be a country specific Spotify issue..
Can confirm that using the workaround @michaelherger supplied with spotty version 4.8.1 solved my issues. -for now at least. 🤔
Thanks Michael!
I've also updated to Spotty 4.8.1 but that didn't solve my issues. ;( I've tried multiple ways of tricking my installation to resolve gew4 to a different IP but so far without success. (Running LMS with Spotty as addon under home assistant)
Did you check the new "fallback" option in the Spotty settings? No need to fiddle with DNS.
@michaelherger No, I didn't even look for a setting. I falsely assumed it "just did it" It works now! Thanks!
I have two things working against me, Not were good at programming and I'm from Denmark. Spottify Connect still doesn't work here ;(.
I have logged into my rasp4 with a ssh. OS is HiFiBerryOS. How do I get from here, to implementing :
"by setting this in /etc/hosts:
104.199.65.124 ap-gew4.spotify.com"
??
@radiorasmus
Type "sudo nano /etc/hosts", and paste in "104.199.65.124 ap-gew4.spotify.com" exit and save and you should be up and running again. You might have to restart the system, not sure.
Hi Mads80,
I think I got it, omitted the "sudo" though. Is there any known and easy cmd that can confirm. Spotify connect went from skipping songs after 1 sec to stopping music after 1 min. It's an improvement ;)
Glad I found this thread. For me without this fix I couldnt play anything through Spotty plugin or librespot for weeks. Still very much an issue for me.
This problem started for me a few days ago in Norway.
After finding this thread, I blocked apresolve.spotify.com
, and now librespot works OK again. For now.
Yeah, got reports from Norwegian users in the past two days, too...
+1 here from Norway
I am using LibreSpot with the JACK Audio backend, and it worked perfectly, but since a few days I can't play any tracks. I didn't change any configurations on my side, so I assume there is a problem with spotify like it already occured here: #510
Setup
Custom compiled with
--no-default-features
and--features jackaudio-backend
running as a user service on a Raspberry Pi.Used Arguments:
LibreSpot allows connections from any device just fine, but can't play any song, no matter of its length. Mots of the time it just skips every song after a few seconds without any audio played, but sometimes it also crashes completely.
Logs: