[JENKINS-31094] Proposal: Jenkins Configuration DSL

[timja]

Problem

Users of Jenkins who use configuration management tools like chef/puppet/ansible want to manage Jenkins and its configuration entirely from those tools. The motivation for this "configuration as code" practice is well understood — see this for one example.

Today some efforts exist, such as Puppet Jenkins module, but according to its maintainers, Jenkins don't behave well for this purpose.

Proposal

My proposal is to take the success of Job DSL plugin and expand the same idea to other configuation of Jenkins. That is, we define a DSL that allows users to configure such things like security realm, slaves, credentials, clouds, and other global configurations.

Such DSL definitions can be placed under $JENKINS_HOME/init.groovy.d so that it gets executed during the boot.

Jenkins used in this mode should also refuse to show the system configuration page, do not write $JENKINS_HOME/config.xml and other global configuration files from Descriptor, and generally do not allow users to make changes in GUI for parts that are coming from the initialization DSL script.

With something like this, Puppet/Chef/Ansible now only needs to lay down files inside $JENKINS_HOME. Combined with Job DSL plugin, the entire Jenkins deployment can be fully managed by code.

---

Originally reported by kohsuke, imported from: Proposal: Jenkins Configuration DSL

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

Epic children:

• 7744

[timja]

kbaltrinic:

Our shop uses Chef to manage our Jenkins infrastructure and yes, current Jenkins is not amenable to this. We are very eager to see progress made here. Based on long experience, I would like to make some recommendations regarding the above and beyond.

1. The idea of providing a configuration DSL makes some sense for those not using chef or puppet. However, chef is a configuration DSL. This proposal makes it sound like I will need to use my chef DSL to code-gen configuration in yet another DSL--something that is bound to be suboptimal. What any chef user really needs is an API that can be 1) queried for current configuration and 2) invoked to change the same.
2. If I am understanding the description correctly, the dsl is only loaded/evaluated at startup time. Rather than solving one of our major issues with Jenkins automated config, this would codify it. It must be possible to reconfigure Jenkins w/o restarting it! Again I think this is something of an artifact of the DSL idea. It seems natural to evaluate a DSL once at startup, but an API is available at all times.
3. While I like the idea of being able to lock down the configuration of Jenkins so that it cannot be done via the UI, this cannot be an all or nothing thing and it can't be automatic. It should be an option that can be enabled but isn't mandatory. A clear example of this is a staging server where we experiment with Jenkins configuration before making it live. The workflow here is apply existing chef cookbooks (configuration DSL). Manually modify and test until we are happy. Codify changes back into DSL/cookbooks. Reapply cookbooks and retest. Lastly apply the new cookbook to production Jenkins.
4. Additionally, (and this may be implied in the above but I want to make it explicit,) any such configuration lockdown needs to be granular. We manage or would like to manage most configuration via the Chef, but some stuff needs to be done on the fly. Obvious stuff is user management and job management. But we also need to be able to manually tweak slave configurations, etc, which are normally managed by chef to handle unusual load situations, etc. Lastly there are always those few configuration changes that can't be tested in staging that we need to be able to apply manually to prod on a temporary basis to assess impact.
5. Lastly, I believe that in order to make this work, there needs to be a design standard for plugins that defines an API-first approach to configuration. Most current plugins are only configurable via their GUI. The occasional plugin provides a CLI extension. Very very few can be adequately configured via groovy. Technically yes they have a groovy API but that API consists of accepting a stapler request. That is unacceptable from an automation standpoint. As this requires enforcing standards against an open source community, it will be probably be the hardest to get right from a non-technical perspective.

[timja]

kohsuke:

My Christmas break fun project that hopes to address this problem

[timja]

batmat:

I certainly support that.
I've been bothered in the past by the idempotency issue myself, but it seems to me this is becoming less and less important with the advent of Docker.
IMO CM tools may still be managing the outer conf and servers, but for example in our setup, we more and more indeed use the Docker module of Ansible to just start prepared images with the right parameters.

So in my use case, my main goal/need is to be able to configure Jenkins by code (through init scripts for example) in a maintainable way. That's also why I prefer to avoid dumping raw XML inside the image (though that works too, already). My Jenkins is gonna be deployed with Docker so I don't really plan to touch it at runtime anyway (I'll see how realistic this is, granted).

I've also had the same idea as the 5. of kbaltrinic: having some standard (compulsory?) extension point to be implemented to accept [re]configuration would indeed be really great. Having recently walked through configuring the Docker plugin through groovy scripting, I'm convinced having an API could have made that more readable.

IMO, we may want to clarify context/expectations here: are we heading to adding a one-shot API, just to simplify the existing groovy way, not designed to be called many times, or a CM-compliant one where we somehow want to describe the "Desired State".
Waiting for more input here, and maybe we should start a thread about that to clarify/set goals .

[timja]

jhoblitt:

My $day_job uses puppet to manage Jenkins deployment, which is invaluable for being able to "test" the ci system. I am a strong "configuration management" proponent and a co-maintainer of the puppet-jenkins module. I have been working on managing jenkins' configuration state by quasi-serializing constructor parameters as JSON snippets passed in and out via the CLI jar and gave a puppetconf presentation on this effort titled Puppet vs. Jenkins. Perhaps needless to say, I have fairly well developed opinions about what makes an application easy to manage from a CM perspective.

I have only briefly looked at the system-config-dsl plugin, so my comments are intended to be of a general nature rather than addressed to a specific implementation. While I can certainly see the utility of a groovy based DSL for complex configuration scenarios, I see this as more of a CM tool unto itself rather than a a good interface for puppet/chef/salt/etc. style CM engine. The fundamental concern I have is that this approach is "configuration as code" rather than "configuration as data". This makes it very difficult to introspect on and modify the configuration state in a meaningful way.

One of the [perhaps poor] bits of advice I give when attempting to explain what makes a good CM integration, is to think about the module as presenting a native configuration API for the application rather than merely restoring state. This stems from the fact that most of the time, the CM end-user doesn't actually want to manage all of the state of an application. Typically, what they want to do is selectively manage state they care about while not disturbing defaults, boilerplate, or values of unknown function that may even change between application versions.

It is, more(1) or less(2), possible to manage the configuration of jenkins by duplicating and resorting the xstream object dumps (1 for the moment, I am ignoring the issue of plugin management, which is its own nightmare; 2 less in that there are often minor changes between plugin versions that require opening the configuration and re-saving it and this completely breaks for encrypted values). The principle issues with the xstream dumps, from the CM point of view, is that the value of internal fields may be mangled in some way that is difficult to reproduce, are difficult for an end user to predict, are unstable (even white-space occasionally changes between core releases), and is it difficult to impossible predict which file a configuration value when end up and what the name of that file may be (eg, the dump file name for plugins), and that the semantics of no two XML documents are the same.

The advantage of a configuration DSL over the xstream serialization, is that it will likely be much easier for an end-user highly familiar with the Jenkins internal APIs to reason directly in it rather than having to dump out a known state and store it away. The downside is, it isn't any easier, perhaps even harder, for a CM engine to express a management API and then dynamically generate the configuration. This means a CM interface will likely have to resort to shuffling opaque files around without any semantic understanding of the contents – that isn't an improvement over doing the same thing with xstream dumps.

IMHO - the best [filesystem based] interface for a CM system is one that uses a well defined data/serialization format, allows the configuration to be split across multiple files in the now common foo.d style, and uses either extremely predictable file names (eg /etc/foo.conf), in that the same configuration values will always be in the same file, or the file name has no significance (eg., /etc/sudoers.d/, /etc/yum.conf.d)

[timja]

seanabbott:

+1 for basically everything above, especially the plugin configuration requirements.

Also, failing some of that, some sort of config validation that is plugin aware (i.e., you can pass it in xml for something you're trying to configure and it comes back with anything that gets ignored or fails) would also be super helpful, mostly for things like jobs and agents. But good config options as outline above might eliminate the need for that.

[timja]

allan:

I have written up using a combination of Chef and the Script Console for a no-xml zero-to-Jenkins setup here [1]. If you know which files and fields in groovy objects to look for state, you can make the whole configuration idempotent even in plain init.groovy.

The approach involves going back and forth the javadoc and plugin unit tests to discover the groovy objects, but I do agree that a DSL can make this approach much cleaner.

[1] http://espinosa.io/blog/2016-03-31-whole-jenkins-in-code.html

[timja]

kohsuke:

jhoblitt, thanks for your thought. The bar of success I set for the system-config-dsl project is to pass the test of folks like you, so hearing from you is very valuable.

When I read your comment, I get the sense that you are trying to point out problems in the project, but all I actually see is the validation. I think this largely stems from the fact that you only see what's already there, and I have some additional pieces in my head that's not yet implemented.

So let me try to explain those additional pieces that are only in my head, and I'm really keen to hear from you again on whether that changes anything from your perspective.

If you leave out the "programmable" portion of DSL and focus on the rest that builds the tree structure, such as the example in README, I see a fairly simple data model that can be seen in just about any tree-capable syntax, such as JSON, YAML, and XML. This in my mind forms the first layer of the configuration file syntax, which you call "configuration as data". And I'm happy to give people those additional "language binding" just in case the particular curly brackets and parenthesis is off-putting to some people.

The programmability of DSL, such as ability to do variable expansions, loop, and so on is then a nice icing for people like me who wants to keep it DRY without relying on external templating tool. In other words, in this case I see "configuration as data" as a subset of "configuration as code."

Compared to "xstream object dump", this immediately wins a few things. One is that users are not constrained by the developers' choice of what information are colocated in one file vs what information are split into separate files. With system-config-dsl plugin, if you want to manage the authentication part but not the authorization part, there's no problem. Likewise you can use two separate files to manage two parts of configurations independently, even when the outcome ends up in the same XML file. You mention /etc/sudoers.d and /etc/yum.conf.d as a positive example, and I'm like, yep, system-config-dsl is solving that.

But ultimately I think the biggest benefit in this approach, compared to "xstream object dump", is that we can now statically generate semantic model of what properties exist on what sections top to bottom, across the entire plugin ecosystem. There is no need for users to be familar with Jenkins internal APIs, and no need for them to look up javadoc. For example, this semantic model can be used to generate a documentation comparable to that of Job DSL plugin, and it can be even used to generate offline semantic checker. You raise the point that one of the problems with "xstream object dump" is that it's difficult to reproduce & predict, and generally reason about. I feel like system-config-dsl is solving that exact problem here.

So in the end, from your conclusion below, in my mind the system-config-dsl plugin provides (1) well-defined, (2) data format, that (3) allows config to split across multiple files. And it solves the plugin installation use cases.

IMHO - the best filesystem based interface for a CM system is one that uses a well defined data/serialization format, allows the configuration to be split across multiple files in the now common foo.d style, and uses either extremely predictable file names (eg /etc/foo.conf), in that the same configuration values will always be in the same file, or the file name has no significance (eg., /etc/sudoers.d/, /etc/yum.conf.d)

So what am I missing here?

[timja]

kohsuke:

I'm just acknowledging here that I'm not making enough progress on this front despite my earlier hope.

I don't want my "claiming" this space to prevent someone else from carrying this flag forward.

[timja]

sag47:

Hi all,
I'd like to throw my 2 cents into this hat . In general, I've found the most effective way to configure Jenkins is to use the Script Console (link) and bootstrap Jenkins and plugins via gradle.

Bootstrapping Jenkins via gradle

Bootstrapping Jenkins via gradle provides something that is currently lacking in Jenkins documentation (which I'm slowly improving over time). The idea of idempotent configurations. That is, if I bootstrap my automation scripts next year they should still be guaranteed to work. The current state of most configuration management "solutions" I've seen is this isn't the case. However, Jenkins publishes the war file and all of it's plugins in Artifactory. This offers a great platform to guarantee idempotent configurations.

Here's an example of my scripts bootstrapping Jenkins and all plugin versions via gradle (link).

Advantages of this method:

• Guaranteed to bootstrap the same way next year as it does this year.
• Maven repository can be mirrored on-site in enterprises to ensure reliable bootstrapping.
• It's easier to spin up test environments with the same versions of Jenkins and plugins this way (even on my own laptop).
• Plugin updates and upgrades can be more safely quality tested and approved for upgrade in production.

"Jenkins State Machine"

When it comes to configuration management and an existing Jenkins installation I've always thought of what I call a "Jenkins State Machine". The Jenkins State Machine is essentially the following:

1. If Jenkins is not in shutdown mode, then configuration changes can occur in the Jenkins runtime.
2. If Jenkins is in shutdown mode, then don't make any configuration changes to Jenkins. i.e. any script which can change the runtime configuration should choose not to change if Jenkins is in this state.
3. If Jenkins is in shutdown mode and there are jobs running, then Jenkins should not be restarted. A "restart" configuration management script should always just choose to wait to allow running jobs to finish.
4. If Jenkins is in shutdown and there are no jobs running, then it is safe to restart Jenkins.

The above state machine can be relatively safely run, automatically, without affecting users too drastically. Something like this can be done without much human intervention as long as the state machine is adhered in all config management scripts.

The above state machine still poses minor problems. Any time jobs queue up during shutdown mode, the state of the queue is discarded on restart. The Jenkins job queue could be preserved when Jenkins is gracefully restarted or gracefully shut down.

Configure via Script Console

One of the most powerful things I've learned about Jenkins and the Script Console is that its runtime is itself the configuration. By updating the runtime, you're updating the configuration and can serialize that configuration to disk with a Jenkins.instance.save() or calling the save() method on other configuration objects.

Pros:

• The "Jenkins State Machine" can be adhered.
• Configuration is immediately updated and the operation is as safe as clicking "Save" in the web UI.
• No restart is required for trivial configuration updates via the Script Console. If the XML configuration on disk is modified then a Jenkins restart is required or executing "Reload configuration from disk". It's an intense and disruptive action. Configuring via Script Console scripts helps avoid this disruption.

Cons:

• Writing Script Console scripts can be labor intensive and is definitely a major automation cost.

In my job, my team and I have paid this up front automation cost for our Jenkins installation. It's amazing how quickly and safely we manage or recover our Jenkins. I plan to share our work in the future.

Some of my automation work around this is already in the wild. jervis_bootstrap.sh (link) shows off how to safely execute script console scripts in Jenkins 2.0 with authentication and CSRF protection enabled. In that same project, the scripts/ directory (link) shows several groovy scripts which are used to automatically configure Jenkins. This includes skipping the Jenkins 2.0 wizard, enabling CSRF protection, and configure other minor bits and bobs. This can be much more comprehensive.

Script Console Scripts are not only great for debugging but configuring Jenkins as well.

I have a video

I presented at a Jenkins Area Meetup and recorded it (link). I explain how I go about writing script console scripts in case others wish to pay this automation cost. After discussing far and wide within the Jenkins community, I'm fairly certain that this approach is unique and effective.

[timja]

mbarr:

So I finally got around to coming back to this, after KK showed me this in May.

I'd say that both Joshua & KK are right-

The config "language" makes life much easier than dealing w/ XML. It's already safe for use in files. The key would be to get something akin to the script console, or an API, to both publish the current state as the DSL, and accept the DSL as an input.

One of the biggest issues out there is dealing w/ security setup, while still letting the CM system manage the tools. Binding a localhost only API that accepted the DSL commands - particularly partial files - would be a great help. And providing the current config as a DSL description would allow (in Puppet speak) a parser to be written for Jenkins as a package type, or as a resource type & provider.

This, plus the Pipeline DSL, makes things much easier.

The fact is: This kind of functionality, implemented as either a DSL, or as a series of REST API endpoints, would make life massively easier for setup & provisioning.

(The key though is to make it possible to access the API via a different, privileged connection that won't break when you turn on normal security)

[timja]

jhill:

In addition to the puppet-jenkins project, there is another initiative which seems to work towards the same goals:

https://docs.openstack.org/infra/jenkins-job-builder/

[timja]

rtyler:

jhill, I don't believe Jenkins Job Builder (jjb) is in this same space. Pipeline, Job DSL plugin, and jjb all occupy a somewhat similar space. Whereas this ticket is a level deeper, about automating the management and configuration of the Jenkins environment itself, rather than the workloads that execute on top of it.

[timja]

sag47:

I've also been taking all of my experience in that script console configuration space and I've created a project to implement some of my ideas as an experiment.

Eventually it will be able to configure the master settings.  I'm brainstorming ways in which it can be dynamically compatible with plugins as well.  Check it out -> https://github.com/sandscape

[timja]

andrey9kin:

Hi!

We have been doing Jenkins configuration through groovy scripts and already accumulated number of the scripts that used by our customers and us internally.

You can check them out here https://github.com/Praqma/JenkinsAsCodeReference/tree/master/dockerizeit/master

Next step for us would be to separate what from how and find a good way to distribute those scripts. Currently, we have a lot of forks of the repo and that is not the best way of distributing updates. We thought about wrapping them into a plugin. Thanks to rsandell who pointed us to config-dsl. ewel and I will try to put some work into config-dsl during the summer.

Is anyone working on it already? Or have wishes, requirements that we should take into account?

BR, Andrey

[timja]

kohsuke:

CloudBees and Praqma are joining hands on this effort, and ewel and ndeloof have started a new project that combines existing efforts. See the mission statement and the goal at JEP #201

[timja]

jhoblitt:

Pardon my ignorance, is this the correct venue to comment on JEP #201 or is there a more appropriate location?