Bug: master removing clients unexpectedly after some time

JayThomason / Tutti

Tutti is a shared music application for the car being made in conjunction with Audi.

3 stars 1 forks source link

Bug: master removing clients unexpectedly after some time #201

Closed JayThomason closed 10 years ago

JayThomason commented 10 years ago

Sometimes the master is unexpectedly removing clients from the jam. This has happened when I let all the phones go to sleep. When I wake up the phones after some time, I find that the master has removed the client(s) and the clients still think they are connected to the jam.

hwray commented 10 years ago

Interesting. So this only happens when the client phones are put to sleep? That must mean that the heartbeat thread isn't running while the phone is asleep?

JayThomason commented 10 years ago

I'm trying to debug this right now. It seems like the master is getting keep alives and the client is sending keep alives so I'm not really sure yet.

JayThomason commented 10 years ago

Wow so this is super weird. What seems to happen from the perspective of the master is this:

receives heartbeats normally
client goes to sleep
- suddenly doesn't receive any heartbeats from client for a while
removes client from jam
suddenly receives heartbeats from client (after this delay)

I think that this has to do with Android managing the threads/CPU really closely when the phone goes to sleep in order to do power management.

JayThomason commented 10 years ago

Interestingly, the server on the phones all respond pretty much instantly to the requests from the clients even if the screen is off. The way to go here might involve using something other than a raw thread + thread.sleep, we might need to use an android construct which actually provides more timely guarantees.

hwray commented 10 years ago

Wow, that is super interesting. Definitely keep me updated with what you find out!

JayThomason commented 10 years ago

It seems like there might be a few options, but I will have to try them and see what works best.

Master doesn't remove clients from jam while screen off.
Master makes period when it checks to remove clients much longer while screen off.
Try to figure out how to use an AlarmManager. This could be a pain because it's intended for usage when you need to post tasks that run very infrequently even if your app is not currently running.
Try using the PowerManager to acquire some sort of lock to keep the CPU from going to sleep. This option could have very negative effects on power usage though.

JayThomason commented 10 years ago

I really want to be able to just not remove clients when the screen is off, but it's kinda unreasonable from a usability perspective. What if the driver is letting everyone else control the application? Then we need to respond to events like users leaving so that all of the other users are aware of the current state of the jam and able to react accordingly.

Therefore I kinda suspect that either 2. or 4. above are our best bets.

JayThomason commented 10 years ago

After thinking a bit more, I think that we're going to have to use the PowerManager or something similar to get this working. What we need to do is ensure that when the screen on a client phone goes off, it can continue to reliably send these keep alive messages to the master without a serious delay.

I guess that the other option we have here is for the master to actively ping the clients with a sort of echo request and check if the request is successful. If it fails then that client should be removed from the jam. Honestly I get the sense that this might be a better way to do it for now.

JayThomason commented 10 years ago

This might also be the reason for this bug as well: https://github.com/JayThomason/Tutti/issues/196

JayThomason commented 10 years ago

Seemingly fixed with changing the architecture.

JayThomason commented 10 years ago

I have also confirmed this works fine when the master's screen is off, one client is actively browsing the music, and another client leaves the jam. The master successfully detects the client has left and propagates that message to the remaining client.