Praqma / pretested-integration-plugin

A generic pretest commit plugin for Jenkins CI
MIT License
7 stars 14 forks source link

Migrate version 3 to a separate new plugin - the untold saga #119

Closed TheLeftMoose closed 6 years ago

TheLeftMoose commented 6 years ago

So let me tell a story.

Ones upon a time we had a team with a jenkins server and some jobDSL... actually a lot of jobDSL. They have really fought great battles to get to the point where the Jenkins server was running well, and only needed some oil in the gears. Then at some point the great almighty god came (played by me...) and wanted to upgrade the jenkins server as we have done lots of times. And every time that was an easy task as the big Docker god helped us make that so easy.

But, with great powers comes great responsibilities and the great almighty god also upgraded all the plugins as always... because that is how we usually do, and I have 5 other jenkins masters to upgrade on the same day. Even the red text giving a warning about changing formats is something that many plugins shows and should sometimes be ignored.

But, low and behold... This one jenkins server stopped working and now the pipeline didn't work. Great efforts was made to find the problem, but only java null pointer exceptions was flooded in the logs. As you can see:

11:29:33 java.lang.NullPointerException 11:29:33 at org.jenkinsci.plugins.pretestedintegration.PretestedIntegrationPostCheckout.perform(PretestedIntegrationPostCheckout.java:140) 11:29:33 at org.jenkinsci.plugins.pretestedintegration.PretestedIntegrationPostCheckout.perform(PretestedIntegrationPostCheckout.java:68) 11:29:33 at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45) 11:29:33 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 11:29:33 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 11:29:33 at hudson.model.Build$BuildExecution.post2(Build.java:186) 11:29:33 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 11:29:33 at hudson.model.Run.execute(Run.java:1749) 11:29:33 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 11:29:33 at hudson.model.ResourceController.execute(ResourceController.java:97) 11:29:33 at hudson.model.Executor.run(Executor.java:429)

By looking into this kind of medieval gibberish, it is no near clear what the problem was. We tried to talk to the IT department (on-topic: https://goo.gl/Nk9XD)

At this point after have tried several things, the almighty god thought about rolling back the entire jenkins server because it looked like jenkins upgrade made the plugin break. The almighty god thought about visiting the great anals where almost forgotten information is forgotten hidden at sight (pip plugin homepage https://plugins.jenkins.io/pretested-integration) it came clear that alongside other relevant information that version 3 is not incompatible with earlier versions. The information in the scripts that you can update "just" upgrade your jobDSL and it should work, though this team don't have imdeiatly time to do this. Also with 1300 lines of jobDSL and multiple SCM systems checkouts the recommendations is not to upgrade according to the scriples. Yes, the team could spend time migrating that chunk to Pipeline, maybe even multi branch Pipeline, but that is not something you do overnight with that amount of code. Though it could be planned if they knew it was coming.

So the almighty God rolled the PIP plugin back and not it all worked as before.

In the land of opportunity the team now have yet another backlog item and a annoying red number in the top of Jenkins when logging in... The team now have to live in fear that no one accidently logs into the plugin administration page and upgrades the plugins marked as upgradable.

One could have thought that the description text that all plugins have could have been a good place to leave just a small comment about this change, though it is not...

In this case the almighty god sits back and reflects at the process and thinks that this team isn't so special. There must be other places around the countries on this flat earth where other teams has the issue.

So, to sum up the story: If you make binary breaking changes to a jenkins plugin, then you fork it and make a seperate new plugin... Please be that guy.

(sorry for the rant...I hope this can open discussion, even if I might be the one that is wrong.)

(and sorry could not find any guidelines for making a bug report, so I found this format the best...)

buep commented 6 years ago

Hi @TheLeftMoose - thanks for your very good story, that is good entertainment to read and very well explain the background and information needed.

I'm sorry you felt the pain of a major bump as unplanned, because we actually took care to avoid it for our users be several means:

As for the wiki page goes, it could probably be more emphasized and to be honest we didn't consider that you would have red warnings on your Jenkins master if you're one of those we recommend not to upgrade. That's a very good point.

Now in your specific case, you're in a group of users where we thought you would be easily okay. You do a lot of the right things like scripting jobs and upgrading your Jenkins master regularly. Our consideration was that all of you who have scripted jobs, should easily be able to migrate to the new syntax as it would only be one or two places you would have to do that if you have architectured your large library of job dsl into smaller methods accommodating re-use. That is typically how we do our job dsl. Even a quick search (maybe replace) could be an approach.

On the other matters of forking the plugin, when breaking backwards compatibility I don't think that is something we will ever do. First and foremost such an action would be a side-effect of something else not working, so we would rather work on the root cause instead. Further it would create a very large number of new plugins in the community if everybody did so. Last, and not finally, I would mean we would, in this case, double our amount of work as we now should maintain or at least bugfix two version of the plugins.

So I apologize for your inconvenience, and I took note of a few things for the future, but we will continue to use semver and warnings in the update center.

Thanks a lot for your feedback.