search

gyptazy / ProxLB

ProxLB - (Re)Balance VM Workloads Across Nodes in Proxmox Clusters. A Load Balancer for Proxmox - and more!
https://proxlb.de
GNU General Public License v3.0
184 stars 8 forks source link

`Bug`: crash when node in maintenance mode is shutdown #107

Open glitchvern opened 21 hours ago

glitchvern commented 21 hours ago

General

When a node is placed in maintenance mode and shutdown or potentially rebooted, proxlb will crash if the node is down during a run.

Weighting

Score: 5

Config

[proxmox] api_host: 172.16.11.25,172.16.11.36,172.16.11.50

api_host: 172.16.11.36,172.16.11.50

verify_ssl: 0 [vm_balancing] enable: 1 method: cpu mode: used maintenance_nodes: dummynode03,dummynode04,avh-proxmox10 mode_option: percent balanciness: 15 ignore_nodes: dummynode01,dummynode02 ignore_vms: testvm01,testvm02 [storage_balancing] enable: 0 [update_service] enable: 0 [api] enable: 0 [service] daemon: 1 schedule: 2 log_verbosity: INFO config_version: 3

Log

<6> ProxLB: Info: [logger]: Logger verbosity got updated to: INFO. <4> ProxLB: Warning: [api-connection]: API connection does not verify SSL certificate. <6> ProxLB: Info: [api-connect-get-host]: Multiple hosts for API connection are given. Testing hosts for further usage. <6> ProxLB: Info: [api-connect-get-host]: Testing host 172.16.11.25 on port tcp/8006. <6> ProxLB: Info: [api-connect-test-host]: Timeout for host 172.16.11.25 is set to 2 seconds. <6> ProxLB: Info: [api-connect-test-host]: Host 172.16.11.25 is reachable on port tcp/8006. <6> ProxLB: Info: [api-connection]: API connection succeeded to host: 172.16.11.25. <6> ProxLB: Info: [only-on-master-executor]: No master only rebalancing is defined. Skipping validation. <6> ProxLB: Info: [node-statistics]: Added node avh-proxmox08. <6> ProxLB: Info: [node-statistics]: Added node avh-proxmox09. <6> ProxLB: Info: [node-statistics]: Added node avh-proxmox10. <6> ProxLB: Info: [node-statistics]: Maintenance mode: avh-proxmox10 is set to maintenance mode. <6> ProxLB: Info: [node-statistics]: Created node statistics. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark13 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark13. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark13. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark11 from API. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark04 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark04. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark04 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark04 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark04. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark06 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark06. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark06 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark06 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark06. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark12 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark12. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark12. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark10 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark10. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark10. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark15 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark15. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark15. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark09 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark09. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark09. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark08 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark08. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark08. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark05 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark05. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark05 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark05 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark05. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-proxlb from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-proxlb. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-proxlb. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark14 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark14. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark14. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark07 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark07. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark07. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM debian-12.6.0-template from API. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark03 from API. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark01 from API. <6> ProxLB: Info: [vm-statistics]: Created VM statistics. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox08 is overprovisioned for CPU by 100%. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox08 is overprovisioned for CPU by 125%. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox08 is overprovisioned for CPU by 150%. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox09 is overprovisioned for CPU by 100%. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox09 is overprovisioned for CPU by 125%. <4> ProxLB: Warning: [node-update-statistics]: Node avh-proxmox09 is overprovisioned for CPU by 150%. <6> ProxLB: Info: [node-update-statistics]: Updated node resource assignments by all VMs. <6> ProxLB: Info: [balancing-method-validation]: Valid balancing method: cpu <6> ProxLB: Info: [balancing-mode-validation]: Valid balancing method: used <6> ProxLB: Info: [balanciness-validation]: Rebalancing for cpu is not needed. Highest usage: 73% | Lowest usage: 61%. <6> ProxLB: Info: [rebalancing-vm-calculator]: Balancing calculations done. <6> ProxLB: Info: [rebalancing-maintenance-vm-calculator]: Maintenance mode for the following hosts defined: {'avh-proxmox10'} <6> ProxLB: Info: [vm-rebalancing-executor]: No rebalancing needed. <6> ProxLB: Info: [cli-output-generator]: Start rebalancing vms to their new nodes. <6> ProxLB: Info: [cli-output-generator]: No rebalancing needed. <6> ProxLB: Info: [post-validations]: All post-validations succeeded. <6> ProxLB: Info: [daemon]: Running in daemon mode. Next run in 2 hours. <4> ProxLB: Warning: [api-connection]: API connection does not verify SSL certificate. <6> ProxLB: Info: [api-connect-get-host]: Multiple hosts for API connection are given. Testing hosts for further usage. <6> ProxLB: Info: [api-connect-get-host]: Testing host 172.16.11.25 on port tcp/8006. <6> ProxLB: Info: [api-connect-test-host]: Timeout for host 172.16.11.25 is set to 2 seconds. <2> ProxLB: Error: [api-connect-test-host]: Host 172.16.11.25 is unreachable on port tcp/8006. <6> ProxLB: Info: [api-connect-get-host]: Testing host 172.16.11.36 on port tcp/8006. <6> ProxLB: Info: [api-connect-test-host]: Timeout for host 172.16.11.36 is set to 2 seconds. <6> ProxLB: Info: [api-connect-test-host]: Host 172.16.11.36 is reachable on port tcp/8006. <6> ProxLB: Info: [api-connection]: API connection succeeded to host: 172.16.11.36. <6> ProxLB: Info: [only-on-master-executor]: No master only rebalancing is defined. Skipping validation. <6> ProxLB: Info: [node-statistics]: Added node avh-proxmox08. <6> ProxLB: Info: [node-statistics]: Added node avh-proxmox09. <6> ProxLB: Info: [node-statistics]: Created node statistics. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark04 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark04. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark04 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark04 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark04. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark11 from API. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark13 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark13. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark13 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark13. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark15 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark15. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark15 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark15. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark10 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark10. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark10 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark10. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark06 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark06. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark06 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark06 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark06. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark12 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark12. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark12 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark12. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark07 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark07. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark07 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark07. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark14 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark14. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark14 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark14. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-proxlb from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-proxlb. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-proxlb found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-proxlb. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark05 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark05. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark05 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark05 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark05. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark08 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark08. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark08 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark08. <6> ProxLB: Info: [api-get-vm-tags]: Got no VM/CT tag for VM avh-benchmark09 from API. <6> ProxLB: Info: [vm-statistics]: Getting disk information for vm avh-benchmark09. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: No (or unsupported) disk(s) for avh-benchmark09 found. <6> ProxLB: Info: [vm-statistics]: Added vm avh-benchmark09. Traceback (most recent call last): File "/app/proxlb", line 1572, in main() File "/app/proxlb", line 1541, in main vm_statistics = get_vm_statistics(api_object, proxlb_config['vm_ignore_vms'], proxlb_config['vm_balancing_type']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/proxlb", line 548, in get_vm_statistics for vm in api_object.nodes(node['node']).qemu.get(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/venv/lib/python3.11/site-packages/proxmoxer/core.py", line 167, in get return self(args)._request("GET", params=params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/venv/lib/python3.11/site-packages/proxmoxer/core.py", line 147, in _request raise ResourceException( proxmoxer.core.ResourceException: 595 Errors during connection establishment, proxy handshake: No route to host - {'errors': b''} ## Meta Please provide some more information about your setup. This includes where you obtained ProxLB (e.g., as a `.deb` file, from the repository or container image) and also which version you're running in which mode. You can obtain the used version from you image version, your local repository information or by running `proxlb -v`. Version: 1.0.4 Installed from: github source installed into container Running as: Container
gyptazy commented 7 hours ago

Hey @glitchvern,

thanks for reporting. I created PR #111 to fix this. This will be part of release 1.0.5.

Thanks, gyptazy