search

kubernetes-sigs / azuredisk-csi-driver

Azure Disk CSI Driver
Apache License 2.0
146 stars 187 forks source link

Expansion of block volume fails #800

Closed bertinatto closed 2 years ago

bertinatto commented 3 years ago

What happened:

The test:

External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works

Fails with:

  Type     Reason                  Age                  From                                                        Message
  ----     ------                  ----                 ----                                                        -------
  Normal   Scheduled               <unknown>                                                                        Successfully assigned e2e-volume-expand-8235/pod-1b9dd0fb-a7de-4f9d-8f37-0796e96a9fd9 to ci-ln-pcprg12-002ac-znwvd-worker-centralus3-gdltz
  Normal   SuccessfulAttachVolume  2m35s                attachdetach-controller                                     AttachVolume.Attach succeeded for volume "pvc-b884879e-bbd7-4853-89a4-1328f58d50e2"
  Warning  FailedMount             38s                  kubelet, ci-ln-pcprg12-002ac-znwvd-worker-centralus3-gdltz  Unable to attach or mount volumes: unmounted volumes=[volume1], unattached volumes=[volume1 default-token-sjszb]: timed out waiting for the condition
  Warning  FailedMount             24s (x9 over 2m32s)  kubelet, ci-ln-pcprg12-002ac-znwvd-worker-centralus3-gdltz  MountVolume.MountDevice failed while expanding volume for volume "pvc-b884879e-bbd7-4853-89a4-1328f58d50e2" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = Could not get size of block volume at path /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b884879e-bbd7-4853-89a4-1328f58d50e2/globalmount: error when getting size of block volume at path /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b884879e-bbd7-4853-89a4-1328f58d50e2/globalmount: output: , err: exit status 1

I believe this code isn't correctly recognizing the block volume.

andyzhangx commented 3 years ago

per this PR(https://github.com/kubernetes-sigs/azuredisk-csi-driver/pull/607), it should work from v1.19, what's your k8s version? @bertinatto

This PR tries to skip node expansion when volume type is block. However, this does not guarantee to work well because volumeCapability is not yet being passed from kubelet until v1.19.

https://github.com/kubernetes/kubernetes/commit/7d6959ce2ce49fa5b37b3aa28d20fdeb01c02223

The root cause for this is the way it get devicePath. I am not familiar with this domain so I dont know how to get the devicepath given volumepath when we cant use the mount information.

[Update] I am using isBlock, err := hostutil.NewHostUtil().PathIsDevice(volumePath) to check if this is a block device and it seems to work.

Testing: Running e2e testsuite offline volume expansion and it worked for block volume.

andyzhangx commented 3 years ago

cc @boddumanohar

k8s-triage-robot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/azuredisk-csi-driver/issues/800#issuecomment-1019401220): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
andyzhangx commented 2 years ago

fixed in v1.13.0 release

AnishShah commented 1 year ago

Does azuredisk support offline expansion?

I'm on 1.25.3 k8s version with 1.22 azuredisk and I see this test failing which was added in 1.25 release (kubernetes/kubernetes#109580).

Kubernetes e2e suite: TestStorageAzureDisk/e2e/[It] External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works 

{Nov 28 17:23:44.290: while recreating pod for resizing: pod "pod-ac9cbfee-5807-431e-9229-7c85918e4f0d" is not Running: timed out while waiting for pod volume-expand-7639/pod-ac9cbfee-5807-431e-9229-7c85918e4f0d to be running failed test/e2e/storage/testsuites/volume_expand.go:242
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4()
    test/e2e/storage/testsuites/volume_expand.go:242 +0xe5a}
andyzhangx commented 1 year ago

Does azuredisk support offline expansion?

I'm on 1.25.3 k8s version with 1.22 azuredisk and I see this test failing which was added in 1.25 release (kubernetes/kubernetes#109580).

Kubernetes e2e suite: TestStorageAzureDisk/e2e/[It] External Storage [Driver: disk.csi.azure.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works 

{Nov 28 17:23:44.290: while recreating pod for resizing: pod "pod-ac9cbfee-5807-431e-9229-7c85918e4f0d" is not Running: timed out while waiting for pod volume-expand-7639/pod-ac9cbfee-5807-431e-9229-7c85918e4f0d to be running failed test/e2e/storage/testsuites/volume_expand.go:242
k8s.io/kubernetes/test/e2e/storage/testsuites.(*volumeExpandTestSuite).DefineTests.func4()
  test/e2e/storage/testsuites/volume_expand.go:242 +0xe5a}

@AnishShah yes, it's supported, and azure disk also supports online resize now, I think you could run following commands to register online resize directly, with the registration, both offline and online resize would work.

az feature register --namespace Microsoft.Compute --name LiveResize
az feature show --namespace Microsoft.Compute --name LiveResize

https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi#resize-a-persistent-volume-without-downtime-preview