jenkinsci / kubernetes-operator

Kubernetes native Jenkins Operator
https://jenkinsci.github.io/kubernetes-operator
Other
596 stars 233 forks source link

Lifecycle management - draining of seedjobs & Backup #889

Closed JCzz closed 8 months ago

JCzz commented 12 months ago

Hi there

We are trying to figure out how the lifecycle is working for the master node and the seedjobs. It looks like the seedjobs pods gets destroyed without any draining when updating the master image.

You can have a look at this github repo( https://github.com/JCzz/jenkins.git ), to see where we are at - and please make a pull request if you can show the way.

And if there is anything we have misunderstud, please let us know.

The baseline for this repo is the jenkins operator documentation. You can just try update the jenkins_instance.yaml with another image like, jenkins/jenkins:jdk11 from jenkins/jenkins:latest. See the jenkins_instance.yaml

The cicd/pipelines/e2e.jenkins contains code that sleeps for 5 minutes, which should be sufficient time to still be running after updating the master image - this is here we would expect the operator to drain the seedjob before killing it.

Tried to add backup keep a state for the operator to handle the lifecycle.

Looking forward and thanks in advance.

Regards Christian

brokenpip3 commented 12 months ago

Hi Chiristian, I'm available to help and give advice but please help me to understand a couple of things, to me is not clear what you are asking for:

We are trying to figure out how the lifecycle is working for the master node and the seedjobs. It looks like the seedjobs pods gets destroyed without any draining when updating the master image.

The seedjob is just a deployment that will manually or automatically trigger the creation of the pipelines by reading a seed jobs repo/directory. When the jenkins image is updated the seed job can goes in clbo 'cause not able to connect to the master but as soon that the master is up the operator will update the seedjob deployment with the new agent secrets and start using it again. Why exactly you need to drain the seedjob deployment? to me look a normal lifecycle, let me know if I'm missing something here

The cicd/pipelines/e2e.jenkins contains code that sleeps for 5 minutes, which should be sufficient time to still be running after updating the master image - this is here we would expect the operator to drain the seedjob before killing it.

The one that you are talking about is not a "seedjob" is the actual jenkins pipeline that will spawn a kubernetes pod that will trigger the pipeline and you were right atm there is no "cleaning service" from the operator pov in the mean time you are updating the master image. You need to manually delete that job. In a normal scenario when the master is always up jenkins will take care of deleting the finished healthy or not healthy k8s pod spawned by jenkins pipelines.

Tried to add backup keep a state for the operator to handle the lifecycle.

what you mean here?

JCzz commented 11 months ago

Hi @brokenpip3

Thank you for you reply, and yes I did misunderstand SeedJobAgent. Just to be sure, and please correct me if I am wrong; the SeedJobAgent's main purpose it to create jobs/projects and look in the repository for new pipeline code when running a job/project?

Question regarding "Lifecycle management":

brokenpip3 commented 8 months ago

Hi @brokenpip3

Thank you for you reply, and yes I did misunderstand SeedJobAgent. Just to be sure, and please correct me if I am wrong; the SeedJobAgent's main purpose it to create jobs/projects and look in the repository for new pipeline code when running a job/project?

yes https://stackoverflow.com/a/37722596

Question regarding "Lifecycle management":

* How would you update the master image, without loosing the existing jobs/projects already running or set up using the SeedJobAgent?

set jenkins in shutdown mode (from manage jenkins), wait for all the jobs to terminate and restart it.