jenkins slave node deleted in jenkins and removed in openstack after master node reboot

jenkinsci / openstack-cloud-plugin

Provision nodes from OpenStack on demand

https://plugins.jenkins.io/openstack-cloud

MIT License

47 stars 83 forks source link

jenkins slave node deleted in jenkins and removed in openstack after master node reboot #256

Open betull opened 5 years ago

betull commented 5 years ago

After jenkins master reboot, all jenkins node that created via openstack cloud plugin deleted in jenkins and all slave virtual machine removed(deleted) in openstack tenant. Although i set to Retention Time is "-1" . When i check jenkins.log, i faced to below logs. How we can fix this issue ?

olivergondza commented 5 years ago

@betull, please attach all relevant logs entries for given server from jenkins log before and after the restart. The behavior does not sound correct, but more information is needed to understand what is happening.

betull commented 5 years ago

jenkins.log this jenkins log has include before and after restart. Also you can find openstack slave cleanup logs.

one slave node config.xml in attached. config.txt

betull commented 5 years ago

Hi @olivergondza same issue occured again. After openstack host closed, Jenkins slave node machines removed automatically in openstack. I will share logs again. How i can slove this issue ? is there any time configuration between jenkins and openstack ? (Although Retention Time is "-1") I tired to create slave nodes and configure again in every time. jenkins.zip

olivergondza commented 5 years ago

After openstack host closed, Jenkins slave node machines removed automatically in openstack.

What do you mean by this? It is expected to delete the Jenkins slave after you have removed OpenStack server manually - it is broken in that case.

betull commented 5 years ago

@olivergondza , No, I mean is different. Openstack host didn't removed manually, just closed (because of any problem ). When started host, other VM's wasn't deleted. Only jenkins slave VM's removed(deleted). Also we have faced same issue, when taken jenkins master VM snapshot. After completed VM snapshot, jenkins slave's removed in openstack tenant.

olivergondza commented 5 years ago

According to the log it is apparent node cinar_box_1.0-5 was rotated as it stopped responding (from Jenkins POW):

INFO: Ping failed. Terminating the channel cinar_box_1.0-5.
java.util.concurrent.TimeoutException: Ping started at 1557828839076 hasn't completed by 1557829079078
        at hudson.remoting.PingThread.ping(PingThread.java:134)
        at hudson.remoting.PingThread.run(PingThread.java:90)
...
May 14, 2019 1:20:56 PM jenkins.plugins.openstack.compute.JCloudsCleanupThread terminateNodesPendingDeletion
INFO: Deleting pending node cinar_box_1.0-5. Reason: Connection was broken: java.io.EOFException

This is normal and expected behavior of the plugin, instead of getting stuck and panicky about the node its connection broke (which might or might not be fixable), it deletes it as it is unusable anyway so new one can be provisioned when needed.

I am sorry but I do not understand your problem definition from the last comment. I do not think there is a way to "close" a VM.