Support concurrent creation of volumes (vmdks)

[hickeng]

Story As a user of VIC I expect to be able to concurrently create containers that use volumes

Details There are two primary issues that are causing undue serialization of volume creation currently: a. a lock in the volume cache b. disk attach/detach reconfigure operations

(a) causes strict serialization and should simply be removed in favour of a singleflight call or similar. (b) causes partial serialization, but the latency can be high enough that its effect is close to full serialization - this can be addressed with batching logic (see #7895)

The state seen by callers of the API should remain consistent, i.e. list should not show a volume that cannot be used or deleted, and only one create for a volume with a given name should succeed.

This impact concurrent container creation performance when anonymous volumes are part of the container specification.

Acceptance

• [ ] add performance test to ensure no regression in volume create after this has merged
• [ ] add CI test for concurrent create of at least 40 containers (SCSI bus limits us to 15 attached at once and there's a concurrent operation limit of 35 so this forces us to exercise queue in both locations).
• [ ] concurrent creation of volume with the same name should have exactly one return success/fail and others should return with already-exists.
• [ ] concurrent use or deletion of volume that's in the process of creation should return "does-not-exist"
• [ ] listing of volumes while volume is being created should not including inflight volume creation.

[zjs]

Outstanding question: what should docker volume ls show during creation?

[hickeng]

what should docker volume ls show during creation?

Yeah, that's the difficult case. The other operations are easy - they block when addressing that specific volume that's in creating.

Any option I considered has to potential to confuse clients as there's no "state" recorded for volumes.

My WIP thoughts are to include it in the list, but block on any use (including delete/inspect) if it's not fully created.