Consistent timeouts(502) on larger playlists

[erricrice]

Hello! I am consistently hitting failures on the 3 retries per chunk when trying to export a large playlist (few thousand songs) - always 502 errors, and never on the same chunk twice. I've been trying to export this playlist a couple of times a day for the last two weeks with no luck and spotify has not been reporting any issues with their API during that time. I've tried this on multiple different ISPs, so I know this isn't an issue unique to my connection.

Since it seems to be an intermittent issue on Spotify's end, you might consider increasing the delay between retries when you hit a failure to give Spotify a second to recover, since even after hitting the first failure it will succeed most of the time on the first or second retry. But then inevitably before finishing the playlist it will hit three in a row and your code exits. Below are a couple of screenshots from the network tab on chrome showing the behavior while exporting a large playlist(the gaps are the 502s that succeeded on a retry).

[watsonbox]

I added a general comment on this topic here before seeing this (identical) issue https://github.com/watsonbox/exportify/issues/142#issuecomment-1545643214:

Thanks for the feedback. I checked on the volumes of these issues and indeed there do seem to be more of them since around April 3rd. The response from the Spotify API is:

{
  "error": {
    "status": 502,
    "message": "Bad gateway."
  }
}

As the Exportify error message explains, 5XX errors are server-side issues which can't be resolved by the client. In this case the 502 error is a generic type of error indicating that the API gateway encountered some unknown error from an upstream service.

The Spotify API does have rate limiting policies which become important when exporting large amounts of data, and returns a 429 error when the client must wait before proceeding. 4XX responses are caused by the client and are usually actionable. Exportify honors these responses, leaving it entirely up to Spotify to indicate how many requests are allowed to be made in any given time window.

That said, it's not impossible that Spotify have begun to respond to large numbers of requests with a 502 response instead of a 429 response, but that's a server-side bug and doesn't suggest that anything can be done on the client to resolve it. I haven't made any code changes to Exportify since last year, well before the increase in this type of error.

The other version of Exportify is still working for me, but that one doesn't allow you to export your liked song or to do a search for playlists.

As this issue seems to be related to the amount of data being exported, it's not surprising to me that a version which exports less data still works. I suppose one possibility might be to add a setting to "Skip large playlists", say 20k or larger. They would then have to be downloaded separately or not at all.

You might also like to try this approach to see if using a separate API application works better.

To summarize: unfortunately, any attempt to resolve a 5XX error is going to be guesswork at best, because the API isn't telling us what went wrong 😞.

Thank you for the debug info showing that the same requests do succeed when repeated 🙏 . As you suggest, I increased the wait time (to 3 seconds) and also increased the number of retries (from 1 to 2).

Let me know if that improves things.

[erricrice]

Hey thanks so much for replying! It's great to see a project that actually gets maintained.

It does appear to be doing what you described now that you deployed the new code, but unfortunately we still ended up hitting the limit. It seems like Spotify is probably doing something intentional here, but obviously doesn't want to expose whatever it is.

I watched it carefully and managed to capture another network screenshot just before the third failure and the page refreshed. Nothing we don't already know unfortunately, it looks like it is just timing out on the same call even with the 3 second gap(twice).

All the chunks are the same size so it seems unlikely that it is a performance problem on spotify's side, but have you tried a smaller chunk size?

[watsonbox]

Hey thanks so much for replying! It's great to see a project that actually gets maintained.

Sure 🙂. I can't claim to be especially responsive but I do get around to these things eventually.

It seems like Spotify is probably doing something intentional here, but obviously doesn't want to expose whatever it is.

Could be, yep 🤷‍♂️.

Nothing we don't already know unfortunately, it looks like it is just timing out on the same call even with the 3 second gap(twice).

Yes I'm afraid I wasn't especially optimistic. As I said before, there's just no indication of what (if anything) to do to fix this on the Exportify side.

All the chunks are the same size so it seems unlikely that it is a performance problem on spotify's side, but have you tried a smaller chunk size?

This got me thinking and I realized that actually a larger chunk size would result in fewer requests to the API. That's typically the way to go when exporting large amounts of data. It turns out they allow 50, so I increased that from 20 as a last shot at this.

[watsonbox]

More people experiencing this using the API here, and no guidance or fix from Spotify since 2021.