Fix deadlock between getSvcRecords and processEndpointDelete

[corhere]

• 20.10 backport of https://github.com/moby/moby/pull/42545
• Fixes https://github.com/moby/moby/issues/46661
• Fixes #2649

We had some hosts with quite a bit of cycling containers that ocassionally causes docker daemons to lock up. Most prominently docker run commands do not respond and nothing happens anymore.

Looking at the stack trace the following is at least likely sometimes a cause to that: Two goroutines g0 and g1 can race against each other:

• (g0) 1. getSvcRecords is called and calls (*network).Lock() --> Network is locked.
• (g1) 2. processEndpointDelete is called, and calls (*controller).Lock() --> Controller is locked
• (g1) 3. processEndpointDelete tries (network).ID() which calls (network).Lock().
• (g0) 4. getSvcRecords calls (*controller).Lock().

3./4. are deadlocked against each other since the other goroutine holds the lock they need.

References https://github.com/moby/libnetwork/blob/b5dc37037049d9b9ef68a3c4611e5eb1b35dd2af/network.go

(cherry picked from commit moby/moby@7c97896747726554165480d102d9e46c54334cba)

[neersighted]

Didn't we already backport this in https://github.com/moby/libnetwork/commit/51413ef1371ffe29ffd341c80574a3bc1c585d90 and https://github.com/moby/libnetwork/pull/2670?

[corhere]

You just fell into the same trap as @thaJeztah. #2670 backported the deadlock fix for processEndpointCreate; this backports the fix for processEndpointDelete

[neersighted]

Aha, just saw https://github.com/moby/moby/issues/46661#issuecomment-1768735326

[cpuguy83]

I'm not sure I understand the description here. It says this is a 20.10 backport but this is merging against master?

[corhere]

@cpuguy83 that's right, moby/libnetwork@master. Ain't 20.10 libnetwork backports fun?

[cpuguy83]

lol, this is libnetwork not moby/moby. Wow I'm quick.