[EKS] [request]: Control over snapshotting when installing EBS CSI add-on

[1999Pra]

Tell us about your request What do you want us to build? The EKS EBS CSI Addon will have a csi-snapshotter sidecar. ebs-csi-controller pod will throw an error of missing CRDs. As there is no requirement for the snapshot functionality CRDs are not installed manually before installing addon.

The requirement is to have a option to disable the csi-snapshotter sidecar while installing EBS CSI Addon or making the addon smarter to notice that CRDs are not installed and avoiding the installation of csi-snapshotter sidecar.

Which service(s) is this request for? This could be EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

The requirement is to have a option to disable the csi-snapshotter sidecar while installing EBS CSI Addon or making the addon smarter to notice that CRDs are not installed and avoiding the installation of csi-snapshotter sidecar.

Are you currently working around this issue? How are you currently solving this problem? N/A

[mikestef9]

Hey @1999Pra is this just an error log that shows in controller logs? Or are you actually noticing issues with the installation and process and functionality.

[mikestef9]

Our understanding is just a spurious log message gets printed that you can safely ignore

E1220 20:48:11.694896 1 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.VolumeSnapshotContent: failed to list *v1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)

But no other side effects visible (other than the inability to use snapshots via CSI obviously) - the containers do not crash or fail, the driver works normally in all other functionality, etc

[ConnorJC3]

Hi everyone, the latest version of the EKS Addon version of the EBS CSI Driver (v1.19.0-eksbuild.2) is being released with the ability to disable the snapshotter sidecar (should be available in all commercial regions within about a week). The schema should allow you to edit sidecars.snapshotter.forceEnable.

This value will default to true on the EKS Addon (for compatibility with prior versions), but can be configured to false in order to disable the snapshotter sidecar if you aren't using it.

[ishworgurung]

Looks like another case of unqualified defaults set by the EKS team. Why is it on by default?

My main gripe is that csi-snapshotter is causing our sonobuoy conformance tests to fail 100% across all our clusters around the globe on production because someone in the EKS team decided one fine dandy morning that defaulting to enabled would be a great idea. It would be great if it happily chugged along(with a sane RBAC default) but as pointed out by https://github.com/aws/containers-roadmap/issues/1919#issuecomment-1360217769 above that's not the case for us as well. Logging should be well controlled; pointless logging is well pointless: disk fill up, log shipping pointlessly, paying for SYN tax needlessly. All this because a saner default wasn't chosen by the EKS team.

Please set it off by default. Those who need CSI snapshot will need to enable it.

[NicholasRaymondiSpot]

We added this JSON to the Configuration Values box found in the Optional Configuration Settings drop-down for the Amazon EBS CSI Driver add-on and it fixed our issue. From the Console UI:

1) EKS, choose your cluster 2) Add-ons tab 3) Select "Amazon EBS CSI Driver" using the bubble option and choose Edit 4) Press the black arrow next to "Optional configuration settings" 5) Scroll down to the "Configuration values" box and paste this in or add it to your existing configuration:

{
  "sidecars": {
    "snapshotter": {
      "forceEnable": false
    }
  }
}

6) Save changes, scale your deployed pods (kubectl scale deployment ebs-csi-controller --replicas=0 -n kube-system, kubectl scale deployment ebs-csi-controller --replicas=2 -n kube-system)

Doing this stopped producing logs & errors related to the csi-snapshotter sidecar. The ebs-csi-controller pod no longer references that container or image configuration.