[BUG] In single node Harvester, endless "unable to schedule replica" is logged in longhorn-manager

w13915984028 commented 2 years ago

Describe the bug

In single node Harvester, endless"unable to schedule replica"is logged in longhorn-manager

In 2 hours, around 2.8M size File, 10,000 lines of message are logged, most of them are related to "unable to schedule replica", "There's no available disk for replica..."

2022-02-24T23:11:46.576967217Z time="2022-02-24T23:11:46Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master state=attached volume=pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1
2022-02-24T23:11:46.577128398Z time="2022-02-24T23:11:46Z" level=error msg="There's no available disk for replica pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1-r-d98cf11d, size 53687091200"
2022-02-24T23:11:46.577169219Z time="2022-02-24T23:11:46Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master replica=pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1-r-d98cf11d state=attached volume=pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1
2022-02-24T23:11:46.577244119Z time="2022-02-24T23:11:46Z" level=error msg="There's no available disk for replica pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1-r-402df7a1, size 53687091200"
2022-02-24T23:11:46.577278519Z time="2022-02-24T23:11:46Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master replica=pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1-r-402df7a1 state=attached volume=pvc-26f59651-ef52-4c7d-8335-08cc5f8223f1
2022-02-24T23:11:46.577360070Z time="2022-02-24T23:11:46Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master state=attached volume=pvc-84f05641-2925-41d4-994b-601dddd30275
2022-02-24T23:11:46.577462942Z time="2022-02-24T23:11:46Z" level=error msg="There's no available disk for replica pvc-84f05641-2925-41d4-994b-601dddd30275-r-a28d6928, size 10485760"
2022-02-24T23:11:46.577474502Z time="2022-02-24T23:11:46Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master replica=pvc-84f05641-2925-41d4-994b-601dddd30275-r-a28d6928 state=attached volume=pvc-84f05641-2925-41d4-994b-601dddd30275
2022-02-24T23:11:46.577543613Z time="2022-02-24T23:11:46Z" level=error msg="There's no available disk for replica pvc-84f05641-2925-41d4-994b-601dddd30275-r-73edb41e, size 10485760"
2022-02-24T23:11:46.577548243Z time="2022-02-24T23:11:46Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master replica=pvc-84f05641-2925-41d4-994b-601dddd30275-r-73edb41e state=attached volume=pvc-84f05641-2925-41d4-994b-601dddd30275
2022-02-24T23:11:47.849117497Z time="2022-02-24T23:11:47Z" level=debug msg="Found 0 node has at least one schedulable disk" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master state=attached volume=pvc-84f05641-2925-41d4-994b-601dddd30275
2022-02-24T23:11:47.849151677Z time="2022-02-24T23:11:47Z" level=error msg="There's no available disk for replica pvc-84f05641-2925-41d4-994b-601dddd30275-r-a28d6928, size 10485760"
2022-02-24T23:11:47.849156617Z time="2022-02-24T23:11:47Z" level=error msg="unable to schedule replica" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=harvester-master owner=harvester-master replica=pvc-84f05641-2925-41d4-994b-601dddd30275-r-a28d6928 state=attached volume=pvc-84f05641-2925-41d4-994b-601dddd30275
2022-02-24T23:11:47.849161687Z time="2022-02-24T23:11:47Z" level=error msg="There's no available disk for replica pvc-84f05641-2925-41d4-994b-601dddd30275-r-73edb41e, size 10485760"

To Reproduce

Steps to reproduce the behavior:

In Single-node Harvester, create any kind of volume (image or other), LH will complain of "unable to schedule replica", those volume has parameter of replia "3" by default.

Expected behavior

Aggregate/Reduce the error message printing, it may overwrite those other useful error messages

Slow down the re-schedule when there is not enough hardware (node) resource (e.g. enque after), in single-node Harvester, it is never possible to have 3 replia. Single-node Harvester / two-node harvester (one management, one worker) are fairly common in developing / testing.

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

Longhorn version: Longhorn latest stable version used in Harvester v1.0.0
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Harvester v1.0.0 master-head
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 0
Node config
- OS type and version:
- CPU per node:
- Memory per node:
- Disk type(e.g. SSD/NVMe):
- Network bandwidth between the nodes:
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
Number of Longhorn volumes in the cluster:

Additional context

Add any other context about the problem here.

PhanLe1010 commented 2 years ago

The idea at https://github.com/longhorn/longhorn/issues/3629#issuecomment-1059652676 maybe a potential solution for this issue

jenting commented 2 years ago

For the single node development, you could change the default replica count from 3 to 1.

Besides that, we could consider changing the message log level from error to warn.

oivindoh commented 2 years ago

@w13915984028 It is possible to have three replicas by settingReplica Node Level Soft Anti-Affinity to true in Longhorn settings. I use this to spread replicas among different drives in my single node setups.

longhorn / longhorn