Fine-grain locking

[pcattori]

Improve performance by refactoring locking to be more fine-grained

Applies to paxos, shardmaster, and shardkv.

[brando90]

@pcattori we have started doing some of the fine-grain locking. I was wondering if we could discuss a little bit what we have so far, how its different from the original locking plan and how its safe (and why we need a different plan). I wanted to make sure that we were convinced that our locking was indeed safe (or decreases the chances of a race condition).

For example, it would be good to identify what things we think could be dangerous because of concurrent writes/reads. For example, we have multiple go routines counting the pings we received. So we need a counter that has safe increment function. Another example could be, one server could receive multiple requests and run multiple handlers. Are we making our acceptors/learners/proposers have inside of them locks so that they don't go to invalid states? What do we mean by invalid states? Is stale data a problem? Is it possible that multiple handlers running at the same time to cause some weird race condition? Why do we need to make map[int]Acceptors etc a ShardMap? If it were not, what problems would we have? Is stale data ever a problem? How are we avoiding these things?

This comment only applies to multi_paxos. We should make different discussion for the other ones (probably), jus tto keep it organized.

[brando90]

@pcattori I would like to document the problem that we spoke of that we had about the crazy fine grained locking that we had earlier, if possible (at the end of the semester probably)