Making idempotency easier

[merlijn-sebrechts]

From email thread: https://lists.ubuntu.com/archives/juju/2017-October/009548.html

Idempotency is a concept new to many charm authors. Many regular sysadmin actions are not idempotent by default. Charm authors need to write a lot of boilerplate to create idempotent handlers.

We need to have more documentation explaining this, giving examples for how to write idempotent code, and we need an "approved" way to deal with the idempotent issue.

This issue has been solved by config mgmt tools years ago, so the preferred way is to leverage an existing tool for that. I don't think charms.reactive needs to reinvent the wheel in this area, as the vision states

charms.reactive is a tool for coordinating config management. It is not a config management tool!

Charms.reactive is used to figure out what state the server needs to be, config mgmt tools are used to actually get the server into that state.

Proposals

Integration with Puppet

I've uses Puppet before in charms, such as in the OpenVPN charm, and I really like it. I also found that Puppet is a lot quicker than Python or Bash, especially for no-ops. Puppet is also used in all Bigtop charms from the Big Data ecosystem, and many other charmers use it so we have a community to support and test an "official" charms.reactive -> Puppet integration. The layer-puppet-base might be a good starting point for this.

A short example of how Puppet is used

This is based on how the OpenVPN charm works.

You create a Puppetfile that describes the "desired state" of the unit (It's a model of the desired state). The following snipped for example defines that NAT forwarding must be enabled.

sysctl { "net.ipv4.ip_forward":
  ensure => present,
  value  => "1",
}

include firewall

firewall { '125 POSTROUTING':
  table    => 'nat',
  proto    => 'all',
  chain    => 'POSTROUTING',
  source   => '{{serverip}}/{{serverslashmask}}',
  jump     => 'MASQUERADE',
}

The handlers fill in the above template, and execute puppet apply to apply that template. The reactive framework is used to figure out what the values should be. Puppet is used to actually apply those configuration values. charm build downloads all the puppet modules and dependencies at build time, thus making sure that your charm will always do the exact same thing.

The disadvantage of Puppet is that it's an entirely new DSL charm authors need to learn. Moreover, modules are written using Ruby, which many charm authors are not familiar with.

Integration with saltstack

Saltstack works very similar to puppet. It has the advantages that it uses yaml for the models instead of a DSL, and modules are written in python instead of ruby. Because of this, it will be more familiar to charm developers and it integrates better with existing charm tooling. The downside is that saltstack is less mature and the community is smaller than Puppet.

An example:

user account for pete:
  user.present:
    - name: pete
    - shell: /bin/bash
    - home: /home/pete
    - groups:
      - sudo

What about Chef and Ansible?

• Chef has a steep learning curve, is very complex, and models are written in Ruby. I've created some Charms with Chef, and it made Charm creation more difficult, not easier. Using Chef underneath Juju also requires you to cripple Chef (use chef-zero) and many community cookbooks stop working when you do that.
• Ansible is easier to understand but it's very focused on orchestration instead of simple config management. Juju + Ansible integration is not a good fit since we'd have to cripple Ansible completely in order to make it run under Juju. Ansible's agentless approach is also not compatible with Juju's agent-centric approach.

[merlijn-sebrechts]

@ktsakalozos might have some insight into how we can make it easier to use Juju + Puppet.

[ktsakalozos]

Hi @galgalesh,

I worked on the bigtop charms that use puppet with @johnsca, @kwmonroe, @petevg and @andrewdmcleod . They might have more concrete ideas how a concrete integration would look like.

Overall Juju+Puppet are a good match. I think we were seeing two main patterns on how puppet was used:

• In the first pattern puppet was doing the initial setup and then juju was taking over to shape the deployment based on the relations available and the config changes. I think this was more of a necessity rather than a design choice. Probably because the puppet scripts were not as flexible as we wanted them to be so we had to "tune" the deployment we got from puppet.
• In the second pattern juju was collecting all the env parameters and was feeding them to puppet. Puppet would re-configuring/re-deploy the system based on the environment presented to it. In this pattern we had to make sure that all the juju configs and relations could be mapped to puppet config params. Spark should be closer to this pattern.

The first pattern is for those who are more comfortable with juju the second for those who have a mature puppet deployment story.

I do not think I have a good answer to your question, this is just my take on puppet + juju.