search

FoxxMD / multi-scrobbler

Scrobble plays from multiple sources to multiple clients
https://foxxmd.github.io/multi-scrobbler
MIT License
299 stars 14 forks source link

Restart an Idle Client #114

Closed Cwavs closed 5 months ago

Cwavs commented 6 months ago

This isn't strictly a bug or a feature request.

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. Client has some sort of failure (e.g downtime)
  2. Mutli-scrobbler tries to scrobble to it but fails 5 times
  3. Client becomes idle
  4. Client doesn't seem to resume activity, and scrobbles continue to queue up

Expected behavior User is presented with a way to re-activate the client and clear the queued scrobbles, or it automatically re-attempts on a new scrobble.

Logs If possible reproduce the issue with debug logging ON

https://privatebin.net/?4a953a85837cbc1f#6vM9m5eYJ6J8YYSzapVqhu93L8a4ydwTpfxzx666q6SS

Versions (please complete the following information): Provide version information for any related sources/clients.

Additional context I've seen this happen a few times recently. When ListenBrainz went down due to a testing mishap, this seems to have caused a ms failure, although I'm not actually sure if it's related because ListenBrainz should still have been accepting scrobbles. The other time is more recently today, where, despite there not being an incident a scrobble failed and a queue built up.

The first time, a reboot did fix the issue, however naturally cleared the scrobbles from the queue which I think is less than ideal, one solution could be to possible save queued scrobbles into a temporary file? I haven't restarted it today just in case it turns out there's something I've missed that can help resume scrobbling!

TheFeelTrain commented 6 months ago

This is happening to me constantly. ListenBrainz seems to lose connection pretty often and requires a restart every time which means losing all of the scrobble backlog.

FoxxMD commented 6 months ago

@Cwavs @TheFeelTrain you can try out MS with manual (re)start for clients by using the foxxmd/multi-scrobbler:develop docker image now.

image

There is still some work to do but it should help for now.

If you have more information or logs related to MS not restarting clients please let me know. MS has a scheduled heartbeat task that runs every 20 minutes that should:

You can see this in privatebin.net/?4a... logs you posted by searching for Heartbeat

Dec 10 18:23:08 mediaserver multi-scrobbler[2502814]: 2023-12-10T18:23:08+00:00 verbose : [Heartbeat] [Clients] Checked Dead letter queue for 2 clients.
Dec 10 18:23:08 mediaserver multi-scrobbler[2502814]: 2023-12-10T18:23:08+00:00 info    : [Heartbeat] [Clients] Attempted to restart 1 clients that were not processing scrobbles.
Dec 10 18:23:08 mediaserver multi-scrobbler[2502814]: 2023-12-10T18:23:08+00:00 error   : [Scrobblers] [Listenbrainz - brainz] Failed to scrobble => undefined {"payload":{"listen_type":"sing...

If the heartbeat is not running OR you are sure the service is available but MS is still returning an error please provide logs of the before and after (heartbeat) where this occurs.

Also as an FYI -- if the scrobbler is running but you still have failed scrobbles you can click the Failed Scrobbles link and then Retry All to force MS to retry scrobbling everything.

image

Cwavs commented 6 months ago

This is good to know. At the time of the logs that I posted, as far as I am aware, LB wasn't having any downtime or issues with scrobbles and I left MS up (without it reactivating) for quite a while until I was forced to restart it to install some updates. Does this sound like an issue with my set up or my home network? LastFM continued to work properly throughout this period.

Unfortunately as it was quite a while back now I don't believe I have any logs saved still from that time, so I'll make sure to keep an eye out for the issue and send any logs your way.

I also think it may still be worth considering store queued scrobbles somewhere so they can persist through restarts, to protect against a longer period of downtime where a server restart might be needed.

As always, thanks for your hard work maintaining this project!

TheFeelTrain commented 6 months ago

I have the logs of it working fine and then suddenly failing. You can ignore the errors about the WebScrobbler, that was this which has already been resolved.

scrobble-2023-12-02.log

This log in particular shows it suddenly fails to scrobble and continues retrying over and over until I restarted the whole container. After the restart (bottom of the log) it works again even though nothing about the ListenBrainz config has changed. You can also see Last.fm and Maloja successfully scrobbling in the middle while ListenBrainz is still failing on the previous one.

Without restarting the container the request will just keep failing over and over with the client set to "Idle" and all of the scrobbles end up queued which you can see in this log from today: scrobble-2023-12-27.log

In fact my last 3+ days of logs is just this repeating since I don't listen to music that often.

I am curious to see if the manual restart will work since it seems like it is trying to restart with the heartbeat occurs. Let me know if you need any more info and I will let you know how it goes when this inevitably happens again.

FoxxMD commented 6 months ago

@TheFeelTrain thanks for the logs that helped me isolate the issue:

I've fixed both of these issues in foxxmd/multi-scrobbler:develop docker image. Please try it out and let me know if it resolves your issue.

TheFeelTrain commented 6 months ago

Now it seems that the Last.fm API doesn't like your changes 😅 scrobble-current.log

FoxxMD commented 6 months ago

Oops! Overlooked something when making these changes. Thanks for the logs, I'll have this fixed soon.

FoxxMD commented 6 months ago

@TheFeelTrain this should be fixed in develop image. Let me know if you have any more issues.

FoxxMD commented 5 months ago

Fixed in 0.6.3