timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-53348] slaves should continue through jenkins restart #10042

Open timja opened 6 years ago

timja commented 6 years ago

It would be nice, and should be possible for slaves to work disconnected through, say, a Jenkins restart.

The ideal scenario of course is a graceful shutdown where Jenkins can tell the slaves it's going to be shutting down but even an ungraceful stop of Jenkins, or the Jenkins machine or a network interruption between them should be survivable.

A slave could, in any of the above events (i.e. being told Jenkins is shutting down, or getting an error writing to the Jenkins server) just switch to writing the stdout/stderr of whatever it is running to a local file and then in the background periodically trying to re-connect to the Jenkins server.

The Jenkins server, through a graceful or ungraceful restart could establish and resurrect the jobs it was running when it was stopped and allow slaves to reconnect and dump their local buffering of it's child's stdout/stderr and then resume logging them to Jenkins.

I am sure the devil is in the details, but it should be doable.


Originally reported by brianjmurrell, imported from: slaves should continue through jenkins restart
  • status: Open
  • priority: Major
  • resolution: Unresolved
  • imported: 2022/01/10
timja commented 6 years ago

oleg_nenashev:

It is definitely doable. E.g. https://www.cloudbees.com/products/cloudbees-jenkins-platform/enterprise-edition/features/long-running-build-plugin (proprietary plugin) does that. https://github.com/jenkinsci/durable-task-plugin/ may also address some use-cases for Master failover, esp. together with Pipeline.

 

timja commented 6 years ago

brianjmurrell:

Interesting.  It doesn't appear however that the Long-Running Build Plugin supports Pipeline jobs, but just freestyle jobs.

The  Durable Task Plug-in also does look interesting. It seems that one won't get the running log of the task though as one is used to but one has to wait for it to complete and then dump it to stdout.

timja commented 6 years ago

oleg_nenashev:

> It doesn't appear however that the Long-Running Build Plugin supports Pipeline jobs, but just freestyle jobs.

Yes AFAICT