shuo-wu
commented
3 years ago I have checked the support bundle (the user linked it to issue #2829 in the email). The attachment looks fine on the Longhorn side. Here is the analysis:
For the RWX volume alertmanager-data :
- Kubernetes requests volume attachment via CSI attacher
2021-08-10T09:58:44.518418375Z I0810 09:58:44.517594 1 connection.go:183] GRPC request: {"node_id":"sky-210015-ds","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":5}},"volume_context":{"diskSelector":"","nodeSelector":"","numberOfReplicas":"3","staleReplicaTimeout":"20"},"volume_id":"alertmanager-data"} - Longhorn CSI Plugin receives attachment call from Kubernetes
2021-08-10T09:58:44.519989195Z time="2021-08-10T09:58:44Z" level=info msg="ControllerPublishVolume: req: {\"node_id\":\"sky-210015-ds\",\"volume_capability\":{\"AccessType\":{\"Mount\":{\"fs_type\":\"ext4\"}},\"access_mode\":{\"mode\":5}},\"volume_context\":{\"diskSelector\":\"\",\"nodeSelector\":\"\",\"numberOfReplicas\":\"3\",\"staleReplicaTimeout\":\"20\"},\"volume_id\":\"alertmanager-data\"}" - Longhorn manager starts to attach the volume then starts a share manager pod. The volume is for the data store. The share manager pod is actually an NFS server. Once the share manager pod is up and running. The RWX volume attachment is considered as successful:
2021-08-10T09:58:49.404877319Z time="2021-08-10T09:58:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Volume\", Namespace:\"longhorn-system\", Name:\"alertmanager-data\", UID:\"6de827c9-4482-49d6-b308-2ae7b919d6d2\", APIVersion:\"longhorn.io/v1beta1\", ResourceVersion:\"21360477\", FieldPath:\"\"}): type: 'Normal' reason: 'Attached' volume alertmanager-data has been attached to sky-210004-ds" 2021-08-10T09:58:44.657043679Z time="2021-08-10T09:58:44Z" level=info msg="Created pod for share manager" controller=longhorn-share-manager node=sky-210004-ds owner=sky-210004-ds pod=share-manager-alertmanager-data shareManager=alertmanager-data state=starting volume=alertmanager-data
The problem is, somehow your cluster spent 4 minutes on starting this share manager pod (from 2021-08-10T09:58:44 to 2021-08-10T10:02:55). There is no error log for this share manager pod on the Longhorn side, which means the share manager workflow runs as expected.
Here is the pod state. No failure or error during initialization:
status:
phase: Running
conditions:
- type: Initialized
status: "True"
lastprobetime: "0001-01-01T00:00:00Z"
lasttransitiontime: "2021-08-10T10:02:05Z"
reason: ""
message: ""
- type: Ready
status: "True"
lastprobetime: "0001-01-01T00:00:00Z"
lasttransitiontime: "2021-08-10T10:02:55Z"
reason: ""
message: ""
- type: ContainersReady
status: "True"
lastprobetime: "0001-01-01T00:00:00Z"
lasttransitiontime: "2021-08-10T10:02:55Z"
reason: ""
message: ""
- type: PodScheduled
status: "True"
lastprobetime: "0001-01-01T00:00:00Z"
lasttransitiontime: "2021-08-10T10:02:05Z"
reason: ""
message: ""
message: ""
reason: ""
nominatednodename: ""
hostip: 172.20.0.2
podip: 10.42.3.220
podips:
- ip: 10.42.3.220
starttime: "2021-08-10T10:02:05Z"
initcontainerstatuses: []
containerstatuses:
- name: share-manager
state:
waiting: null
running:
startedat: "2021-08-10T10:02:06Z"
terminated: null
lastterminationstate:
waiting: null
running: null
terminated: null
ready: true
restartcount: 0
image: longhornio/longhorn-share-manager:v1_20210416
imageid: docker-pullable://longhornio/longhorn-share-manager@sha256:c0528766fa0fd8c21632e76b0a79844fa484b3f58c53c855ac4575f4076abad1
containerid: docker://c40ef6a3b5a25665e5a3a19af96bcdb2241027324cdd3d69cc587ebcfbf63208
started: true
qosclass: BestEffort
ephemeralcontainerstatuses: []
qdupuy
Describe the bug Launching a workload that requires a new RWX volume takes a long time (4~5 minutes). The direct cause is the unexpected slow RWX volume attachment. Here is the original issue report: https://rancher-users.slack.com/archives/CC2UQM49Y/p1628589991044200?thread_ts=1628252452.025500&cid=CC2UQM49Y
To Reproduce Steps to reproduce the behavior:
Expected behavior The RWX volume can be attached in 5~30 seconds typically.
Log
Environment:
Additional context Add any other context about the problem here.