[BUG]Spotify Connection fails (API Limit?)

[AHub88]

Is this a bug in companion itself or a module?

• [X] I believe this to be a bug in companion

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

The connection to spotify fails after 2-3 days. The status is green and "OK", but you can no longer control spotify. It only works again if you create a new app in the Spotify Developer Dashboard. I could imagine that it is due to the many requests and Spotify limits them.

Steps To Reproduce

No response

Expected Behavior

No response

Environment (please complete the following information)

OS: Windows 10
Browser: Chrome 94
Companion Version: 2.2.0-b177f8fb-3722

Additional context

No response

[Julusian]

It appears that accounts created after May 2021 have lower api limits https://developer.spotify.com/community/news/2021/05/27/improving-the-developer-and-user-experience-for-third-party-apps/

So it is possible that you are hitting those limits. That said, in surprised that the line really spikes up on the 9th and stays at almost double.

Also I am surprised that it breaks completely after a few days, and requires being reconfigured again. Does exiting companion for a few minutes and starting it again have the same effect?

[AHub88]

I‘ve tested it with two companion instances. I think that‘s the reason for the doubling.

It works sometimes when I restart companion and wait a few minutes. But not for a long time

[sytone]

I am seeing a lot of traffic as well. Hitting 350000 requests per day which... seems a lot.

[bevanjkay]

@sytone that is about right. The module is configured to poll Spotify for status every 250ms

[Julusian]

Yeah I think I have seen this happen before too. Something to do with the retries or the way the status is polled can cause it to get stuck in an almost infinite loop of polling.

This was something I was addressing in #29, by making a 'workqueue' for the polling, which ensured that there was only ever one request in flight at a time and would also limit the rate a bit. I havent finished that PR because the spotify connection library doesnt handle timeouts, so when a request times out it gets stuck waiting forever. I have been running this branch for a few weeks and observe this happening every couple of days..

[sytone]

I have pulled the module in as a volume for my docker container and updated frequency to 5000ms and it seems to be behaving better and every 5 seconds for update is more than enough for me.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Julian Waller @.> Sent: Tuesday, February 22, 2022 2:12:31 PM To: bitfocus/companion-module-spotify-remote @.> Cc: Sytone @.>; Mention @.> Subject: Re: [bitfocus/companion-module-spotify-remote] [BUG]Spotify Connection fails (API Limit?) (Issue #30)

Yeah I think I have seen this happen before too. Something to do with the retries or the way the status is polled can cause it to get stuck in an almost infinite loop of polling.

This was something I was addressing in #29https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbitfocus%2Fcompanion-module-spotify-remote%2Fpull%2F29&data=04%7C01%7C%7Ca5eab727cc7142b6877708d9f6506e36%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637811647578987847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=oxA1N%2FMLviobwfqdzok84NtxCdCi7PXyGijBHi3g4No%3D&reserved=0, by making a 'workqueue' for the polling, which ensured that there was only ever one request in flight at a time and would also limit the rate a bit. I havent finished that PR because the spotify connection library doesnt handle timeouts, so when a request times out it gets stuck waiting forever. I have been running this branch for a few weeks and observe this happening every couple of days..

— Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbitfocus%2Fcompanion-module-spotify-remote%2Fissues%2F30%23issuecomment-1048261759&data=04%7C01%7C%7Ca5eab727cc7142b6877708d9f6506e36%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637811647578987847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=zzeAaznb2ZeHw33h6jSHEGkowgef%2BfRqRcGy8M7CXZc%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAKVVE7LY7QZVJT426SKQATU4QC47ANCNFSM5OL7VOGA&data=04%7C01%7C%7Ca5eab727cc7142b6877708d9f6506e36%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637811647578987847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wwlfzQi6jDdGNQ0BRzWcAHnS4LwdGqR4y3wqkpFmQ7I%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[AHub88]

Is there any solution? Unfortunately the module is currently not usable.

[joezimjs]

So is the Spotify documentation wrong? https://developer.spotify.com/documentation/web-api/guides/rate-limits/ The only rate limits it mentions are 30-second rolling limits. If it works for a couple days then stops (and then comes back about a month later in my experience), then it seems like a different limit is the problem.

In any case, it seems like adjusting that 250ms interval to something slower is an acceptable workaround for now, but it might be a good idea to listen for the 429 responses that documentation I linked to talks about. Instead of just firing on an interval, just set a timeout for the next one after each response, using the set time you come up with if it comes back as 200 but setting it long enough to clear the 429 error is one comes back.

I'd love to try to implement it myself, but I'm not sure how to edit the one installed on my Companion software nor how to install a version I've created. If I can get instructions for how to do that from someone, I might try making a PR.

[joezimjs]

Before I get into a lot of details that many of you may not want to read, I wanted to note a workaround that my church is using right now. We created a button that toggles the connection instance between enabled and disabled, and it's green while enabled and gray while disabled. This way, the Spotify module isn't running all day and causing these rate limits to be hit. We only have it running while we're using it.

Now, on to the word vomit!!

Ok, I created a little proof of concept for running this outside Companion to test it out: https://github.com/joezimjs/companion-module-spotify-remote/commit/9367d25fa0b07ae96fb3223eb92a6883aa3ff551

Note: I created a file called run.js that you just run with Node. I commented out some lines that referred to built-in Companion stuff in module.js and created stubs for some of the other things in run.js.

I left it running overnight and this is what I got in the console:

Note: I ran the test with a 500ms interval, not the 2000ms you see in the commit.

It's just logging out the play/pause state, so that's why you see "paused" and the number before that is the number of times in a row that that message was logged, So, with over 23,000 polling requests, I got 3 unknown errors. That's no big deal, it's just polling, so we can recover from that with the next request. The big thing was that after all those requests, suddenly there was a 429 response from Spotify saying we need to wait over 18 HOURS before the next request. So, apparently there's a daily rate limit as well, which I believe is around 34,000 based on the graphs that we see for this app:

Note, I had done a lot of testing with super low intervals before I ran this overnight test to make sure I was handling the 30 second rate limit nicely. Considering you could fire off 345,600 requests in one day with the current 250ms interval, there definitely seems to be a per day limit, which we can hit in less than 2.5 hours with the current interval. This then gives you a 21.5 hour window of time where you can't make any requests, so it appears like the spotify module just never works anymore, but you're just never using it within the tiny window of time where it's working each day.

So, potential solutions time; we need to stay under the daily limit - which means if it's running all day - we need to average about 23 requests per minute or less. So, we can just set the interval to 3000ms (or higher) and you'd be left with 5000+ more requests before you hit the limit that you can use for actually controlling Spotify.

In any case, I think the polling time needs to be lengthened and I don't think 1000ms will really be too slow for anyone. Every time you click to actually perform an action, it immediately polls for the status again afterward, so it's not like you have a severely delayed response to an action. And due to this, I don't think just changing it to a 3000ms interval is terrible option despite it seeming like a cheap workaround.

Alternatively, if we want to keep the polling interval relatively short, we can keep track of the last time someone performed an action. After 10 minutes without any input, we can slow the interval down to once a minute or something. And after an hour or more with no input, switch it to once every 5 or 10 minutes. This seems like a good idea, but all the time limits are arbitrary and may not work well for other people.

So, in the end, I just think changing the interval to 3 seconds or more is the simplest and most foolproof way of fixing the issue.

[Julusian]

@joezimjs thanks, that is some good detective work there. Yeah, I think that changing the polling time to 3s would be a good quick fix, to get the module usable for all. Changing the polling that slow will no doubt cause some other bug reports to appear of that the timings dont update smoothly, but that is something that can be improved later. I do think we should also listen to 429 responses, but that is a much larger task, and I wouldnt want to start on that until I have finally finished up #29

[Julusian]

I've changed this to 3000ms in the latest beta.

Leaving this issue open as a reminder to myself to look at making the handling more intelligent in some way

[joezimjs]

As for listening to 429 responses, if you look at my commit and ignore the console logs, the commented out/deleted bits that use the companion instance skeleton, and the extra run.js file, you should be able to copy that over for handling 429 responses.

[joezimjs]

Hmm... I guess I read the graphs wrong. I read it as around 35,000 but I missed a digit: they're actually saying 340,000, which means, it's including all requests, even ones that come back with a 429. I'm going to run some full days tests with the 3K ms and get back with results to see how long it lasts...

[joezimjs]

Ok, after running for over 24 hours without any rate limits hit, I think we're safe with 3 seconds. 👍

[AHub88]

Yes, I've tested it too

[Julusian]

In case 3s is not ok, this is now a config field