Closed kiwik closed 5 years ago
Reproduce just now
stack@ubuntu1604 ~ $ openstack --os-cloud openlab-packet server list
+--------------------------------------+-----------------------------------------+--------+---------------------------------------------+----------------------------------+-------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------------------------------+--------+---------------------------------------------+----------------------------------+-------------+
| ff9609b2-6a5f-4ade-8af8-dc2bc8f68672 | ubuntu-xenial-packet-openlab-0000005064 | ACTIVE | openlabzuul-net=192.168.1.24, 147.75.38.167 | openlab-ubuntu-xenial-1552901589 | zuul-flavor |
| e7695127-aabd-4fa3-a724-7f6085405b21 | ubuntu-xenial-packet-openlab-0000005009 | ACTIVE | openlabzuul-net=192.168.1.21, 147.75.38.174 | openlab-ubuntu-xenial-1552901589 | zuul-flavor |
+--------------------------------------+-----------------------------------------+--------+---------------------------------------------+----------------------------------+-------------+
stack@ubuntu1604 ~ $ ping 147.75.38.167
PING 147.75.38.167 (147.75.38.167) 56(84) bytes of data.
64 bytes from 147.75.38.167: icmp_seq=1 ttl=43 time=312 ms
64 bytes from 147.75.38.167: icmp_seq=2 ttl=43 time=320 ms
^[[A^C
--- 147.75.38.167 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 312.267/316.469/320.672/4.239 ms
stack@ubuntu1604 ~ $ ping 147.75.38.174
PING 147.75.38.174 (147.75.38.174) 56(84) bytes of data.
From 147.75.198.243 icmp_seq=3 Destination Host Unreachable
From 147.75.198.243 icmp_seq=1 Destination Host Unreachable
From 147.75.198.243 icmp_seq=6 Destination Host Unreachable
From 147.75.198.243 icmp_seq=5 Destination Host Unreachable
From 147.75.198.243 icmp_seq=8 Destination Host Unreachable
From 147.75.198.243 icmp_seq=9 Destination Host Unreachable
From 147.75.198.243 icmp_seq=7 Destination Host Unreachable
From 147.75.198.243 icmp_seq=10 Destination Host Unreachable
From 147.75.198.243 icmp_seq=11 Destination Host Unreachable
From 147.75.198.243 icmp_seq=12 Destination Host Unreachable
^C
--- 147.75.38.174 ping statistics ---
14 packets transmitted, 0 received, +10 errors, 100% packet loss, time 13074ms
pipe 4
stack@ubuntu1604 ~ $
@mrhillsman we have confirmed this issue and disabled packet pool now, would you contact with packet operator to resolve this issue?
Don't face the issue when we complete periodic pipeline refactor https://github.com/theopenlab/openlab/issues/173 .
Thank you for reporting bug!
Describe the bug
This issue only happen in Packet public cloud. Packet instance is launched by OpenLab Nodepool, and show it as available node in pool, then the instance may be unreachable after a period of time, if OpenLab Zuul dispatch a job to the instance, will cause job
RETRY_LIMIT
failing.We know Nodepool will check whether node is available by SSH host checking after launching it, if can't access, Nodepool will delete the instance, try to launch new one. So we guess Nodepool checking is OK, but instance can be accessed after a period of time.
Zuul job log in OpenLab
Instance 147.75.38.151 is unreachable, but other nodes are OK
openstack console log show
Related Job information
NA
Steps to reproduce the issue
Describe how to reproduce the issue: Probabilistic reproduction, try to ping
ready
packet nodes in OpenLab nodepool, sometime you can find unreachable node.Additional context
Add any other context about the problem here.