DeadLock with LeaderChange/StateChange Events

[nemosupremo]

The setState function locks the state mutex that isn't unlocked when the event handlers are called. As a result if an event handler calls server.State() in response to a StateChange or LeaderChange event, the application can deadlock.

Example gist: https://gist.github.com/nemothekid/8576383

I think this can be fixed by either unlocked the mutex before the events are called, or moving the event callbacks inside a defer

[nemosupremo]

Actually the more I look at it, the more I'm seeing that there are problems with my code. Maybe its because the documentation is still nascent but if you try to Do(command) anything as a result of event callback, you can end up in a deadlock. Its actually obvious once you read the code and it may be my fault for not being properly tuned with the library.

This may be a bad idea, but what if event callbacks ran in their own goroutine (or pass events over channels)? It seems that currently, that care has to be taken when working on the raft goroutine from outside the libary, else you might end up in a deadlock.

[xiang90]

@nemothekid You can use the event.Value and event.PrevValut instead of calling the server.X to inspect the changed variables for now. Your comment are reasonable. We can work around it.