Closed bdevcich closed 4 months ago
We need to consider the same for lusre-csi-driver. There might be other areas for this as well.
@behlendorf, I'm making changes to set this to a sane default for our 3 daemonsets:
maxUnavailable
can be set to a number of nodes/pods or a percentage. Setting it to 100%
would cause it to attempt a restart on all the nodes at the same time, 50% would do half, 25% a quarter, etc. Do you have a preference on what percentage (or hard number) should be?
This value will be adjustable for each system.
@bdevcich as an initial swag how about 25%. This seems like it may be a reasonable compromise between propagating the changes rapidly and potentially overwhelming the system / container repository / other. Then we can tune on a per system basis, and revisit the default as needed.
@bdevcich as an initial swag how about 25%. This seems like it may be a reasonable compromise between propagating the changes rapidly and potentially overwhelming the system / container repository / other. Then we can tune on a per system basis, and revisit the default as needed.
Perfect, that's the percentage that I've been playing around with in my testing.
PRs here to default these all to 25%:
For nnf-dm, the NnfDataMovementManager resource is edited rather than the DaemonSet directly. The manager is responsibe for managing the DaemonSet.
@behlendorf I am comfortable closing this issue after we implemented this manually today on El Cap. Do you agree?
Yup, things are looking much better after these changes.
From @behlendorf:
When you add/remove a
lustrefilesystem
resource, thennf-dm-manager-controller
sees that and then adds/removes aVolume
andVolumeMount
to thennf-dm-worker
DaemonSet
. Kubernetes then handles it from there and restarts thennf-dm-worker
pods on each rabbit to mount/umount that filesystem change. TheDaemonSet
defines this for the updateStrategy:That
maxUnavailable: 1
is what is causing the sequential behavior. We'll need to tweak this.