search

kubernetes-sigs / vsphere-csi-driver

vSphere storage Container Storage Interface (CSI) plugin
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/index.html
Apache License 2.0
295 stars 181 forks source link

ability to inject kubelet path better in the csi driver #793

Closed sandaymin123 closed 1 year ago

sandaymin123 commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug kind feature

What happened: On custom K8 deployments on VMWARE, a different kubelet path is used other than the default /var/lib/kubelet but /var/data/kubelet instead. As a resullt, the default value is passed to the csi node driver to be used to mount the storage /var/lib/kubelet fails to perform the mounts as it does not have access to /var/data... -->

Describe alternatives you've considered editing deamonset, -->

What you expected to happen: By way of a config map or other overriding means, (env var,etc), the ability to inject the kublet path directory.

Describe alternatives you've considered

How to reproduce it (as minimally and precisely as possible): Configure a K8 cluster with a different kublet path other than /var/lib/kubelet

Anything else we need to know?:

Environment:

@dims @jsafrane and @Mike Fedosin

dims commented 3 years ago

@sandaymin123 what was your workaround?

sandaymin123 commented 3 years ago

edited the deamonset

SandeepPissay commented 3 years ago

@sandaymin123 I see that you edited the daemonset to workaround this issue. Do you have a generic solution for this problem that can be used for both regular k8s and custom k8s? And can you provide more details on the custom k8s deployment you are talking about and why the default /var/lib/kubelet path does not work for the custom k8s deployment?

sandaymin123 commented 3 years ago

@SandeepPissay The motivation for using a separate path is as follows:

We allow customers to order infrastructure that can have a larger secondary disk so it can be used by the container orchestration runtime for pulling images, creation of empheral storage, and encrypt the disk that could house customer sensitive data. Customers running very large workloads (i.e Spark or Data and A&AI) often run of of space when just using a single disk for both root partition and /var/lib/kubelet path are co-located

Varoius Solutions like RH OCS let you set a proprerty on a configmap to change the path..Also, the AZURE CSE Driver also lets you have that option on it's helm chart (example...https://github.com/kubernetes-sigs/azurefile-csi-driver/tree/master/charts) I don't want to be prescriptive on how to change the driver to specify the property as there are different ways of getting that input. cc: @dims

SandeepPissay commented 3 years ago

Thanks @sandaymin123 for the context. I'll add this feature request to our internal backlog. I think we can take care of it when we have an operator for the vSphere CSI.

tejohnson123 commented 3 years ago

@SandeepPissay Hi. I'm not sure the time frame for the operator, but if it's a ways out yet, would you be open to providing something simple like a patch file or kustomization?

SandeepPissay commented 3 years ago

@tejohnson123 I would prefer to have the operator provide this option and also tested in the context of the operator. Feel free to customize our yamls for the time being if that works for you.

k8s-triage-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

tejohnson123 commented 3 years ago

We would still like this option to be part of the operator installation when that is available. Keeping this issue open.

/remove-lifecycle rotten

jvrahav commented 3 years ago

@sandaymin123 If you dont mind, can you answere few queries can you share which values in the daemon set did you modify. Also, was this a custom kubelet or the one from upstream. Is this case only when we use a cloud provider that is not the vsphere cloud provider or would this case be valid regardless of cloud provider.

tejohnson123 commented 3 years ago

@jvrahav

We modified the pod-mount-dir volume mount:

        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /var/data/kubelet <-------------- This
          mountPropagation: Bidirectional
          name: pods-mount-dir
        - mountPath: /dev
          name: device-dir
        - mountPath: /sys/block
          name: blocks-dir
        - mountPath: /sys/devices
          name: sys-devices-dir

and

volumes:
      - hostPath:
          path: /var/lib/kubelet/plugins_registry
          type: Directory
        name: registration-dir
      - hostPath:
          path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
          type: DirectoryOrCreate
        name: plugin-dir
      - hostPath:
          path: /var/data/kubelet <----------- this
          type: Directory
        name: pods-mount-dir
      - hostPath:
          path: /dev
          type: ""
        name: device-dir
      - hostPath:
          path: /sys/block
          type: Directory
        name: blocks-dir
      - hostPath:
          path: /sys/devices
          type: Directory
        name: sys-devices-dir

We use upstream kubelet. This would be the case regardless of cloud provider.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

tgelter commented 2 years ago

/remove-lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

tgelter commented 2 years ago

/remove-lifecycle stale

poblin-orange commented 2 years ago

This is really a pain. Seems that vdo operator has this ability. A supported helm chart for vsphere-csi-driver would be helpfull

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/793#issuecomment-1362718715): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.