[BUG] Longhorn V2 disks are added successfully, but show webhook errors in the UI, and in BD status conditions

[tserong]

Describe the bug When adding an additional disk to a node with provisioner set to Longhorn V2, the disk is added successfully, but later shows admission webhook errors, for example:

After rebooting the host, we see a different error:

...and the corresponding block device status shows the AddedToNode condition is false, even though the disk is provisioned successfully:

# kubectl get bds -n longhorn-system -o yaml
[...]
  spec:
    [...]
    provision: true
    provisioner:
      longhorn:
        engineVersion: LonghornV2
  status:
    conditions:
    - lastUpdateTime: "2024-10-04T07:36:57Z"
      message: 'Internal error occurred: failed calling webhook "mutator.longhorn.io":
        failed to call webhook: Post "https://longhorn-admission-webhook.longhorn-system.svc:9502/v1/webhook/mutation?timeout=10s":
        proxy error from 127.0.0.1:9345 while dialing 10.52.0.174:9502, code 502:
        502 Bad Gateway'
      reason: Error
      status: "False"
      type: AddedToNode
    [...]
    provisionPhase: Provisioned
    state: Active

To Reproduce Steps to reproduce the behavior:

1. Add a new disk to a host using the Longhorn V2 provisioner.
2. Wait and observe the errors described above.

Expected behavior The disk is added successfully and we don't see any errors.

Environment

• Harvester ISO version: master as at 2024-10-02
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): virtualized

Additional context I thought I'd fixed this with https://github.com/harvester/node-disk-manager/pull/142, but apparently I didn't :-/

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• [x] Where is the reproduce steps/test steps documented? The reproduce steps/test steps are at: this PR

• [ ] Is there a workaround for the issue? If so, where is it documented? The workaround is at:

• [x] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)? The PR is at: https://github.com/harvester/node-disk-manager/pull/152

  • [x] Does the PR include the explanation for the fix or the feature?

• [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?
  • The automation skeleton PR is at:
  • The automation test case PR is at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#1587