Puppet 7 Support

[QueerCodingGirl]

Does choria support / work with puppet 7? Every documentation i could find mentions puppet 6, with the exception of the "Puppet 7 Support" of the choria/mcollective_agent_puppet version 2.4.1 module mentioned here: https://choria.io/blog/post/2021/03/29/april_releases/ and even for this module the forge.puppet.com still shows the "<= 6" constraint.

Slightly related to maybe-not-updated-docs: I saw a docker image on the docker hub, but can't find anything about a docker deployment in the docs under "Deployment". Is there a docker / docker-compose example running choria?

Maybe its only outdated and/or missing documentation, i can't tell.

[ripienaar]

It does support Puppet 7 yes, it's probably we just didnt update the metadata.json for some of the modules - a common mistake. But it works in Puppet 7, there's one thing mco puppet resource that doesn't work - appears to be some internal behavior changes in Puppet and I cant figure it out, otherwise its fine.

Regarding docker, no there isn't a docker deployment option. It's used in the Helm charts though at github.com/choria-io/helm - but I dont officially support the Helm based deployment yet (though it works, its how I run the central chora infra)

[QueerCodingGirl]

Cool! Thank you for the fast response! If theres Puppet 7 support then this site needs an update too: https://choria.io/docs/deployment/requirements/

If i would now go on and open an issue for a docker deployment option... are there any chances you would not close it with a "wont-do" label? ;)

[ripienaar]

If you want, there's a choria-io/website project where you can send a documentation update, else I can also.

Docker deployment for the central component - broker, etc - or for servers? I can see the value in the first of course but the problem is there's a bunch of implied dependencies. For example, we need a CA, which CA would that docker deployment use? If you have Puppet, yeah I guess that can be used. So I guess the devil is in the details.

I am not against supporting more modern methods of course (see Helm, where we do solve the CA issue via cert-manager) but for the typical Puppet user I think it woldnt be widely used.

[QueerCodingGirl]

I think you are way faster than i am. The time it would take me to make the PR with the one line update to the docs vs. you doing the change...

I am currently running my complete puppet 7 stack inside docker. I am using the "pupperware" project from puppetlabs. I would just need an example of the central components (broker etc.) with a minimal config in a docker-compose file format and then i could include that in my pupperware setup easily.

[ripienaar]

ok, well, the gross way is to run puppet inside docker to configure that instance of the broker, perhaps during a packer build of the base? That at least does not invalidate the documentation for you - since it would all just work.

Though thinking about it - as the container is aimed at the k8s work - it does not run the broker as root. Though of course also easy to show you a config of a puppet configured broker.

Start up https://github.com/choria-io/vagrant-demo and grab it from the puppet node

[ripienaar]

(updated docs, next build around next release will publish it)

[QueerCodingGirl]

Thank you for your support! I will try the container made for k8s and look up a config example from the vagrant-demo. Might post a docker-compose example here when i got it running successfully here.

[ripienaar]

plugin.choria.broker_network = true
plugin.choria.network.listen_address = ::

# conection between  broker if >1
plugin.choria.network.peer_port = 5222
# list of brokers, all brokers get the same list
plugin.choria.network.peers = nats://n1.example.net:5222,nats://n2.example.net:5222,nats://n3.example.net

thats about it really for the basic use case

[ripienaar]

https://github.com/choria-io/puppet-choria/blob/main/templates/broker.cfg.epp is the puppet template that also sets up logging and such

[QueerCodingGirl]

I have given up on getting this running and playing nice with docker / pupperware and am using the choria puppet module now. I have some trouble with that too, but at least there are some docs about it to look at. ;) For the problem i am having right now i will open another issue as i believe it has nothing to do with puppet 7 support but with me having an error in my configuration. I am closing this one now.

[ripienaar]

fwiw

generate certificates for your broken by name lets say choria.example.net using puppetserver ca

Prepare a configuration directory as here ensuring files are owned by user 2048 - or readable 2048:

$ tree -u
.
├── [2048    ]  broker.conf
└── [2048    ]  ssl
    ├── [2048    ]  ca.pem
    ├── [2048    ]  cert.pem
    └── [2048    ]  key.pem

1 directory, 4 files

broker.conf has:

plugin.choria.broker_network = true
plugin.choria.network.listen_address = ::
plugin.choria.network.public_url = choria.example.net:4222

plugin.choria.ssldir = /etc/choria/ssl
plugin.security.provider = file
plugin.security.file.certificate = /etc/choria/ssl/cert.pem
plugin.security.file.key = /etc/choria/ssl/key.pem
plugin.security.file.ca = /etc/choria/ssl/ca.pem
plugin.security.file.cache = /etc/choria/ssl/cache

And you're good to go

$ docker run -ti --rm -v `pwd`:/etc/choria -p 4222:4222 choria/choria:latest broker run --config /etc/choria/broker.conf
INFO[0000] Choria Broker version 0.20.2 starting with config /etc/choria/broker.conf
INFO[0000] Starting Network Broker
ERRO[0000] SRV lookup for _mcollective-broker._tcp failed: could not find your facter command
INFO[0000] Starting new Network Broker with NATS version 2.2.0-beta.58 on :::4222 using config file /etc/choria/broker.conf  component=network
INFO[0000] Starting nats-server                          component=network_broker
INFO[0000]   Version:  2.2.0-beta.58                     component=network_broker
INFO[0000]   Git:      [not set]                         component=network_broker
INFO[0000]   Name:     c0066e4753a9                      component=network_broker
INFO[0000]   ID:       NC3P6JOFANMMB4VKUGT7HREXQX53O2KENEKWXLBD53ROJ2XXPDWMZTV6  component=network_broker
INFO[0000] Listening for client connections on [::]:4222  component=network_broker
INFO[0000] TLS required for client connections           component=network_broker
INFO[0000] Server is ready                               component=network_broker
INFO[0000] Cluster name is CHORIA                        component=network_broker
INFO[0000] Listening for route connections on [::]:5222  component=network_broker
INFO[0000] Statistics gathering disabled, set plugin.choria.stats_port

At this point you're good to go, double check with nc from elsewhere

$ nc choria.example.net 4222
INFO {"server_id":"NCOFKSWHF3PZU6K4455KI5CWY5MMP4GCXGH7VLI5ARYORVDC3GOSQO3X","server_name":"a2006e9cd421","version":"2.2.0-beta.58","proto":1,"go":"go1.15.7","host":"choria.example.net","port":4222,"headers":true,"auth_required":true,"tls_required":true,"tls_verify":true,"max_payload":1048576,"client_id":2,"client_ip":"172.17.0.1","cluster":"CHORIA"}

make sure host is reported correctly

[QueerCodingGirl]

Why should this be set to file and not to puppet?

plugin.security.provider = file

[ripienaar]

When set to Puppet it tries to find the various certificates and all in the usual Puppet locations /etc/puppetlabs/puppet/ssl and specifically uses the fqdn to figure out those names. So while I didnt try that I ancitipated this will probably just not work, besides from no puppet in the container.

So by putting the files in a known location I can just go ahead and tell it where to find them rather than rely on bunch of clever Puppet related defaulting

[ripienaar]

You can of course mount in a puppet directory with things in the right place - probably also need working facter then, but anyway containers often dont have hostnames that would help.

[QueerCodingGirl]

Thanks for the explanation! I have that part set up working with "puppet" (and without docker)

My current problem is that, despite getting good output on the "mco choria show_config" command, i do not get a response from the node(s) back when running "mco tasks run choria::ping -v --message teeeest" or when running "mco puppet status --dm=choria -v". I only get this warning:

warn 2021/10/08 16:39:49: client.rb:283:in `rescue in start_receiver' Could not receive all responses. Did not receive responses from local-hostname-where-puppet-and-choria-are-running.domain.tld

[ripienaar]

hmm, so a few things to try:

I assume you confirmed your nodes are connecting to the right place, right broker etc? Lets check one of the nodes you are expecting to respond.

Here we should see connected_server matches your container based broker.

$ cat /var/log/choria-status.json|jq
{
  "identity": "node.example.net",
  "uptime": 1170,
  "connected_server": "nats://broker.example.net:4222",
  "last_message": 0,
  "provisioning_mode": false,
  "stats": {
    "total": 0,
    "valid": 0,
    "invalid": 0,
    "passed": 0,
    "filtered": 0,
    "replies": 0,
    "ttlexpired": 0
  },
  "certificate_expires": "2026-09-12T12:21:38Z"
}

If that's right, check the logs - maybe even set a node to debug for a bit - then do choria ping, your missing node should log:

{"agent":"discovery","component":"server","identity":"node.example.net","level":"info","msg":"Handling message c7ccb05a9cb54d359f3be8ebc7ce95a2 for discovery#ping from up=rip","subsystem":"agents","time":"2021-10-08T17:04:37+02:00"}

And if thats fine, lets see choria ping --debug, if that works, we need a mco ping also after setting /etc/choria/client.conf to debug

[QueerCodingGirl]

As said before, everything choria related is NOT running inside a docker container, but directly on the host, configured via puppet module / hiera. Also good to know: I am running everything mentioned here on this single host, even the node thats not responding is this local host. I am doing this to avoid the firewalls as a possible source of my headaches.

"connected_server": "nats://puppet:4222",

its not the puppet fqdn but it does resolve to the same ip.

"identity": "puppet.domain.tld",

is set to the puppet fqdn.

choria ping and mco ping do both work fine and give me the ms it took to ping.

[ripienaar]

ok, thats great then basics are good - so here the 4222 is the broker container right?

So then we need to see what logs you get when you run your mco puppet status, the --dm choria configures it to talk to PuppetDB, I assume you did that, but lets start without the --dm choria to isolate things

[QueerCodingGirl]

Still no container, but yes, the choria broker is listening on 4222. "mco puppet status" gives me exactly the same output as "mco puppet status --dm choria":

mco puppet status

| [ > ] 0 / 1warn 2021/10/08 17:26:20: client.rb:283:in `rescue in start_receiver' Could not receive all responses. Did not receive responses from puppet.domain.tld

Summary of Applying:

 No aggregate summary could be computed

Summary of Daemon Running:

 No aggregate summary could be computed

Summary of Enabled:

 No aggregate summary could be computed

Summary of Idling:

 No aggregate summary could be computed

Summary of Status:

 No aggregate summary could be computed

Finished processing 0 / 1 hosts in 22001.69 ms

No response from:

puppet.domain.tld                                   

[ripienaar]

OK, so we have to see logs from`choria.log

[QueerCodingGirl]

Command and Output:

mco tasks run choria::ping -v --message teeeest
Retrieving task metadata for task choria::ping from the Puppet Server
Discovering hosts using the choria method for 2 second(s) .... 1
Attempting to download and run task choria::ping on 1 nodes

warn 2021/10/08 17:34:12: client.rb:283:in `rescue in start_receiver' Could not receive all responses. Did not receive responses from cmp-puppet01.domain.tld

No response from:

    cmp-puppet01.domain.tld                                    

Could not download the task choria::ping onto all nodes

/var/log/choria.log excerpt:

{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:31:49+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:31:58+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:08+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:18+02:00"}
{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:32:19+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:54992 - cid:6 - \"v1.12.3:go:puppet.domain.tld\" - Client Ping Timer","time":"2021-10-08T17:32:22+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:54992 - cid:6 - \"v1.12.3:go:puppet.domain.tld\" - Delaying PING due to remote ping 2s ago","time":"2021-10-08T17:32:22+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:28+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:38+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:48+02:00"}
{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:32:49+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:32:58+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:08+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - Client connection created","time":"2021-10-08T17:33:09+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - Starting TLS client connection handshake","time":"2021-10-08T17:33:09+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - TLS handshake complete","time":"2021-10-08T17:33:09+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","time":"2021-10-08T17:33:09+02:00"}
{"component":"authentication","level":"debug","msg":"Registering user \"\" in account \"choria\"","time":"2021-10-08T17:33:09+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - \"v0.6.2:ruby2:cmp-puppet01.domain.tld\" - Client Ping Timer","time":"2021-10-08T17:33:12+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:18+02:00"}
{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:33:19+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:28+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:38+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:48+02:00"}
{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:33:49+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:33:58+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:34:08+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:37604 - cid:22 - \"v0.6.2:ruby2:cmp-puppet01.domain.tld\" - Client connection closed: Client Closed","time":"2021-10-08T17:34:12+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:34:18+02:00"}
{"component":"server","identity":"puppet.domain.tld","level":"debug","msg":"Writing server status to /var/log/choria-status.json","time":"2021-10-08T17:34:19+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:54992 - cid:6 - \"v1.12.3:go:puppet.domain.tld\" - Client Ping Timer","time":"2021-10-08T17:34:22+02:00"}
{"component":"network_broker","level":"debug","msg":"10.228.30.135:54992 - cid:6 - \"v1.12.3:go:puppet.domain.tld\" - Delaying PING due to remote ping 2s ago","time":"2021-10-08T17:34:22+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:34:28+02:00"}
{"level":"debug","msg":"Starting NATS /varz update","time":"2021-10-08T17:34:38+02:00"}

[ripienaar]

I'd have expected some server related logs there - I am guessing you set server.conf to debug and not broker.conf right? We want to see debug from the server.

Maybe also show choria inventory your.node --no-facts

[QueerCodingGirl]

broker.conf is set to debug and to log into the file i posted here.

i believe i found the reason of my problem, but i have no idea where i could fix it. when you look carefully at my posted command outputs you notice 2 different hostnames: puppet.domain.tld and cmp-puppet01.domain.tld Thats not a censoring / copy-paste mistake from me. One is the hostname of that machine (cmp-puppet01...) and the other (puppet.domain.tld) is the FQDN i set up everything under. The FQDN also has its own ip address.

When i try to run "choria inventory puppet.domain.tld --no-facts" it works and when i use cmp-puppet01.domain.tld it fails with this message:

FATA[0004] Could not run Choria: no inventory received

[ripienaar]

All right and if you try to run your task using -I fqdn of a working one then it works?

I am afk atm happy to just jump on a zoom next week and help you figure it out. Probably easiest. I am keen to get some docs/instruction up for that kind of deploy

[QueerCodingGirl]

Wow, this command works. mco tasks run choria::ping -I puppet.domain.tld -v --message test I really do not understand why. Shouldn't the -I parameter just filter what nodes i am targeting and when i do not use this parameter it should just select all nodes?

Thank you very much for your help so far! Have a nice weekend!

[ripienaar]

Ok. Then instead of -I then I bet —dm=mc also works.

So what’s happening is there is confusion between what puopetdb thinks the fqdn is and what the node and therefor choria thinks it is.

Discover against puppetdb finds one host name and the node will ignore requests for different host names.

[QueerCodingGirl]

Yes, the --dm=mc flag works too. I can't find "mc" in the docs, what is it for?

Would it be possible to get a config option in the choria server.conf like "ssl_alt_subjects_i_should_also_listen_to"? [1] That would enable me to just configure the other FQDN as another thing this host would listen and respond to. It would not affect people that do not have this option set (backwards-compatible) and i only have to set it on hosts that have multiple FQDNs.

[1] ofc thats not the exact name i would give that option ;)

[ripienaar]

The --dm=mc is the default discovery method where it asks the fleet who is there and then access them by reported names, this avoids whatever confusion is between your puppetdb and your actual machines.

Regarding server/broker, please see, I think we have some terminology confusion https://choria.io/docs/concepts/architecture/

I think you mean the broker needing alt names well, it doesnt request its own cert - it just uses the cert already there, so its up to you to use Puppet methods for configuring the certnames and alt names and more. But thats not the problem here.

The problem is your node has one FQDN and Choria is configured with another name, you need to pick one.

[QueerCodingGirl]

There is no confusion, i meant the choria-server. I have a server on the same node as the broker, because i want to manage this node as well as others.

I do not think that i need to pick one, because the infrastructure shouldn't conform to the tools, but the tools to the existing infrastructure. I have this setup with multiple names on one machine already present and for reasons that would go too far to explain here, that wont change any time soon. It would be great if i could still manage this node with choria, i just need a way to tell choria about this situation.

[ripienaar]

You can set identity=whatever.you.like in its config

[QueerCodingGirl]

That setting is already set the whole time we were testing. I just tried leaving it out or setting it to the other name, it did not work. I can see the other name in the output but the node does not respond in any case while using --dm choria Maybe the identity setting is not used/considered everywhere inside the code?

[ripienaar]

Choria is oppinionated about this, its probably not going to just let you change it like that because people kept messing this up and relates to picking of the certificates by PuppetDB

Your certnames must match your FQDNs - the default

https://choria.io/docs/deployment/requirements/

So your only option - maybe - is to set MCOLLECTIVE_CERTNAME=other.fqdn

[ripienaar]

(you have to set that prior to starting choria-server)

[ripienaar]

for me setting identity works - as in it tries to find a cert in puppet directory matching identity

[QueerCodingGirl]

MCOLLECTIVE_CERTNAME did not work either.

I am giving up on this. I guess i wont use choria-server on this node and it has to stay a pet i have to admin manually.

Thank you for your time!

[ripienaar]

All right. Offer to get on a zoom still stands fwiw