[JENKINS-53767] Offer plugin management tooling

[timja]

Jenkins does offer a web UI to manage plugin installation from a live master instance, but in many deployments administrator would like to control installed plugins by version and by tooling, before jenkins master starts.

• docker official image provides an install-plugins.sh script which evolved far beyond its initial "help script" scope.
• configuration-as-code implemented plugin installation hack based on PluginSite.Plugin#deploy. (Removed as of v1.8 of JCasC plugin)
• open-source update center does expose plugins-version.json on casc request so one can manage versions for all hosted plugins, not just latest
• custom war packager does support plugin installation on his own
• jenkins evergreen does manage plugin installation based on updates from metadata server, using his own logic implemented by nodejs evergreen client.
• jenkinsfile-runner does manage plugin installation implemented in Go
• coreOS provides a groovy script to handle required plugins
• cloudbees DEV@Cloud uses a set of Chef Ruby recipes for the same purpose
• etc

 

Everybody is re-inventing the wheel, partially implementing the "details" of plugin management (signed metadata, artifacts checksums, plugins detached from core, ...). It becomes obvious Jenkins should provide adequate tooling for plugin installation outside a live Jenkins instance. 

---

Originally reported by ndeloof, imported from: Offer plugin management tooling

• assignee: stopalopa
• status: Resolved
• priority: Major
• resolution: Fixed
• resolved: 2020-12-12T08:42:23+00:00
• imported: 2022/01/10

Epic children:

• 4427

• 4957

[timja]

oleg_nenashev:

Thanks for submitting it ndeloof! I totally agree we should streamline it at some point and offer a single solution

[timja]

jstrachan:

I'm a big fan of GitOps and using source code to define what version of jenkins core, dockerfile, plugins and their versions etc that folks wanna use. Then to upgrade a plugin/core its a Pull Request.

It'd be nice to enumerate all the different ways folks configure jenkins core + plugins + versions via source code to generate/setup their Jenkins server.

e.g. there's the classic Dockerfile + plugins.txt approach like this...

https://github.com/garethjevans/jenkins-quickstart01/tree/master/jenkinsConfig/departmentFoo/teamA

or there's Custom War Packager style:

https://github.com/garethjevans/jenkins-cwp-quickstart01/blob/master/packager-config.yml#L11

then there's CasC YAML too.

 

It shouldn't be too hard to handle those 3 source code formats.  

Then we could create a plugin for updatebot https://github.com/jenkins-x/updatebot/ to automate the generation of PRs for folks using GitOps to upgrade jenkins and/or plugins using either Evergreen,  incrementals:  https://repo.jenkins-ci.org/incrementals/ or releases: https://repo.jenkins-ci.org/releases/

 

[timja]

ndeloof:

Part of the problem here is that those tools rely on the Update Center layout to bake download URL for version of plugins to install, while this layout is not defined by a spec.

So we need

1. define|document the plugin update center mechanism (metadata signature, how to retrieve hosted plugin versions and download URL, ...)
2. offer an implementation that can be used in various installation context

For (2) I'd like we extract PluginManager into a standalone executable jar so

•  it can run as a CLI jar and been used in docker images as a replacement for install-plugins script
• it can be used by custom war packager as an utility library to implement the same need
• the exact same bits are used by jenkins PluginManager (to enforce consistency)

[timja]

batmat:

FWIW, CWP and Evergreen are using (no enforcement/checker yet, so there's probably smallish discrepancies) both use the Bill Of Materials JEP-309: https://github.com/jenkinsci/jep/tree/master/jep/309.

So we already have the standard format defined. So we should all use it. And if by chance it's deemed to miss some things, then JEP-309 should just be amended so that we agree on something. I agree with ndeloof this subject of handling Jenkins dependencies is becoming wasteful, fluffier and fluffier with all these variants, and should become a tool on its own so that people stop rewriting the (square) wheel everywhere.

I've thought about all this just a bit and I wonder technically this should be:

• I find a tool like this would be good to be usable everwhere easily using CLI, so writing it in Go seems a good direction
• at the same time, Jenkins ecosystem and consumers would probably need it in Java. Ideally, in that case, even Jenkins core associated logic could be extracted (cannot judge of the associated complexity though) so that really everyone use the same resolver tool. A CLI usable packaged version of this tooling (fatjar or whatever) could then be used:
  • in the Docker image builds
  • in Evergreen generation of resolved ingest.json from the essentials.yaml
  • from Jenkins-X's updatebot
  • from CWP
  • ...

Like for Maven dependencies, we need a tool able to:

• compute a dependency tree without (or with as few as possible) downloads
• download a whole dependency tree (including optional dependencies or not, here lies some combinatorial too).

[timja]

jglick:

There is already a tool which understands how to update versions of components including incremental versions from a specified branch (such as origin master); as per JENKINS-51929, it would “just” need a shim to also read/write the JEP-309 format. That is only going to be useful, however, if the file is actually listing all the plugins you are bundling, as discussed in JENKINS-53506.

[timja]

oleg_nenashev:

What if we propose it as a GSoC 2019 project idea? If we do so, would somebody be interested to mentor it?

[timja]

batmat:

I'd be ok to co-mentor this subject.

[timja]

oleg_nenashev:

FTR https://jenkins.io/projects/gsoc/2019/project-ideas/plugin-installation-manager-tool-cli/ 

[timja]

casz:

In JCasC we ended up removing the plugin management support as it was causing issues on startup. See https://github.com/jenkinsci/configuration-as-code-plugin/pull/769

Thank you oleg_nenashev for the GSoC write up

[timja]

casz:

From JCasC point of view:

Should be possible to add an optional JCasC configurator to Jenkins core.
Reading the configurations at startup and invoke the installation of plugins before ever starting to configure plugins.
That would at least take care of plugin installation where JCasC is concerned.
Though it would still require more plugin installation tooling.

Regarding plugin installation tooling

I still strongly believe that tooling should either stay within Java or a mix of Java and Go.

If it was pure Java I could foresee the tool becoming overly large and burden by dependency hell.

A mix of Go and Java could work.

Go would be the main source library and has a better native HTTP client.

Java jar would just be a wrapper for calling go functions: https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf

[timja]

ndeloof:

> Should be possible to add an optional JCasC configurator to Jenkins core.

if added to Jenkins Core then JCasC can't be a plugin anymore.

 

My initial idea was to refactor PluginManager (and siblings) so they can be used as a standalone library for plugin management/installation, then can be either embedded into another tool, or invoked from command line with the adequate `-cp` option pointing to this lib within jenkins.war/WEB-INF/libs.

I'm also fine with a Go implementation which would make perfect sense for portability in various scenarios. In both cases, a well defined JEP on plugin management in Jenkins would help, as today we have to reverse-engineer the existing codebase to understand plugin management and update-center mysteries.

 

[timja]

oleg_nenashev:

Just an update here, we will have a GSoC project for it this year: https://summerofcode.withgoogle.com/projects/#5741119520899072

The format is yet to be defined, but it will be handled under the Platform SIG umbrella.

 

 

[timja]

nickbrown:

This might help address the following ansible jenkin plugin management problem: https://github.com/ansible/ansible/issues/24864

[timja]

fwilhe:

Hi,

I like the idea of this project quite a lot, cool stuff.

I have one potential use case where I don’t find a ticket matching it.

For context, I work at SAP and we have a Jenkins image based on upstream with a custom plugins.txt file. Currently we use latest of all plugins. What I would like is to feed that plugins list into the tool and get a now up to date plugins.txt with all dependencies resolved and versions fixed, so we can use that to build a reproducible image. So, instead of a directory where the plugins are installed, my desired output is a “resolved” plugins.txt file. Is this in scope for this project? I think most required code is already there, and thus implementation should not be too much work (looking at PluginManager.java from a birds eye view).

 

Kind regards

 

Florian

[timja]

jglick:

JENKINS-47498 might be a better input source, as the list of plugins would actually be mechanically vetted, not merely the latest on offer.

[timja]

timblaktu:

Can someone in the know please add a comment about how and where this work was resolved?

[timja]

timja:

https://github.com/jenkinsci/plugin-installation-manager-tool