kickstart script fails with icinga-2.9 on linux (debian)

jschanz commented 6 years ago

Expected Behavior

kickstart script should run

Current Behavior

see https://github.com/Icinga/icinga2/issues/6516 kickstart script fails with

> bash icinga2_kickstart.sh
check: icinga2 installed - OK
ERROR: $ICINGA2_USER has not been defined
...
+ ICINGA2_NODENAME=...
+ ICINGA2_CA_TICKET=...
+ ICINGA2_PARENT_ZONE=...
+ ICINGA2_PARENT_ENDPOINTS=...
+ ICINGA2_CA_NODE=...
+ echo -n 'check: icinga2 installed - '
check: icinga2 installed - + icinga2 --version
+ echo OK
OK
+ '[' '4.4.12(1)-release' ']'
+ RHEL_SYSCONFIG=/etc/sysconfig/icinga2
+ DEB_SYSCONFIG=/usr/lib/icinga2/icinga2
+ '[' -f /etc/sysconfig/icinga2 ']'
+ '[' -f /usr/lib/icinga2/icinga2 ']'
+ ICINGA2_SYSCONFIG_FILE=/usr/lib/icinga2/icinga2
+ . /usr/lib/icinga2/icinga2
++ ICINGA2_ERROR_LOG=/var/log/icinga2/error.log
+ '[' '' ']'
+ fail '$ICINGA2_USER has not been defined'
+ echo 'ERROR: $ICINGA2_USER has not been defined'
ERROR: $ICINGA2_USER has not been defined
+ exit 1

due to no set variables in /usr/lib/icinga2/icinga2

Possible Solution

maybe nothing if https://github.com/Icinga/icinga2/issues/6516 is fixed, otherwise the kickstart script needs a fix

Steps to Reproduce (for bugs)

install icinga-2.9
export kickstart script from director
start kickstart script

Your Environment

icinga2 - The Icinga 2 network monitoring daemon (version: r2.9.1-1)
director-1.4.3

dnsmichi commented 6 years ago

@lippserd some thoughts on the script here.

https://github.com/Icinga/icingaweb2-module-director/blob/master/contrib/linux-agent-installer/Icinga2Agent.bash duplicates certain things:

the SSL_DIR variable ensures that the certificate directory is created. This is already done by the setup wizard CLI commands in Icinga and does not need manual changes. So, sourcing ICINGA2_USER from the environment is not necessary at all.

https://github.com/Icinga/icingaweb2-module-director/blob/master/contrib/linux-agent-installer/Icinga2Agent.bash#L39

trusted-master.crt is a trust file where you actually provide proof to pki request (and node setup) that you trust the parent node you're querying for a certificate. This is automated here, and can be put into a temporary location, no need for the actual path to the TLS certificates.
I'm wondering why the script uses pki request instead of node setup. The latter provides everything configurable in a transparent way.
The CLI commands which invoke icinga2 don't have any error checking, so if save-cert fails, you'll never know why pki request actually breaks.
Please note that /etc/icinga2/pki for TLS certificate paths was available in < 2.8, which is out of support by now with 2.8 and 2.9.
. "${ICINGA2_INSTALL_PREFIX}/lib/icinga2/prepare-dirs" "${ICINGA2_SYSCONFIG_FILE}" shoudn't be needed at all, that's handled by the daemon's systemd service file / sysvinit script.

TL;DR - the sysconfig file for the Icinga daemon is for user customizations, not for reading the default paths and what not. This was wrong in previous Icinga 2 versions, and now has been fixed with 2.9.

Thomas-Gelf commented 6 years ago

Most of this script is a community contribution, and it is around since 2.4 or 2.5. I'm sure every tweak in there has some historic reasoning behind it. What we need is a script that works with multiple Icinga versions. It's there to configure Agents, so I'd really love to still support 2.6. That's for example what Debian Stable ships. If that's a problem we might drop 2.6 support for 1.5, but at least 2.7 should definitively still be supported.

/etc/icinga2/pki has been kept as it works for all versions, even 2.8 can be kickstarted with that path, didn't try 2.9. I prefer a warning over a broken setup, so this seemed to be a safe choice
we could of course behave differently for different versions. @dnsmichi if that's the way to go please let me know how to determine the Icinga 2 version in a fail- and upgrade-safe way for automation
what assumptions can be made for $DAEMON, how does such a script find the binary on every Linux system for the involved versions?
it must be possible to run the script regardless of whether the Daemon has already been started once. This means that directories could be missing. I roughly remember that there have been related issues. Users, directories, prepare-dirs and such are there to make sure nothing breaks because of missing directories. Any guidance on how to assure this on 2.6+ would be highly appreciated.
did node setup always have a non-interactive mode?

If all this means to rewrite the whole script that's fine for me. If it means to ship different implementations (IF/ELSE/CASE) based on the Icinga version that's also ok, we can then drop outdated ones as soon as Director stops supporting them as Agents. It would help enormously if you could provide some guidance addressing above questions, I wasn't able to keep up with all related changes in the core.

Thanks, Thomas

NB: How do our "environments" fit into this picture?

dnsmichi commented 6 years ago

/etc/icinga2/pki has been kept as it works for all versions, even 2.8 can be kickstarted with that path, didn't try 2.9. I prefer a warning over a broken setup, so this seemed to be a safe choice

icinga2 still starts and copies the certificates from the old paths, but the CLI commands in 2.8+ expect /var/lib/icinga2/certs as the true source. It would still work, but honestly the new paths are safe and tested, and we kind of expect that when someone upgrades from 2.8 to 2.9. We could have removed the deprecated options from the ApiListener in 2.9 but decided to keep it for longer to avoid breaking setups.

we could of course behave differently for different versions. @dnsmichi if that's the way to go please let me know how to determine the Icinga 2 version in a fail- and upgrade-safe way for automation

Since you want older versions being involved too, we can only use what's there. icinga2 --version is one shot, including some sed/awk/cut magic. Or you go with package listings and extracting the version from there, similar to what we are sometimes used to with Puppet.

what assumptions can be made for $DAEMON, how does such a script find the binary on every Linux system for the involved versions?

icinga2 is expected to be in $PATH, if not, the user did something wrong, compiled by themselves, or uses an unsupported platform. The entire documentation expects that you just run icinga2 object list and not /usr/sbin/icinga2 object list.

it must be possible to run the script regardless of whether the Daemon has already been started once. This means that directories could be missing.

The only CLI command which would start another daemon, is icinga2 daemon and this generates a human readable error. One thing you need for icinga2 object list and icinga2 variable is a precompiled cache file from a config validation for example.

I roughly remember that there have been related issues. Users, directories, prepare-dirs and such are there to make sure nothing breaks because of missing directories. Any guidance on how to assure this on 2.6+ would be highly appreciated.

It doesn't hurt but shouldn't be a requirement. One would need to test that, might be a good project for trainees @dgoetz

did node setup always have a non-interactive mode?

Yes, it was designed with automation in mind. The interactive CLI command is node wizard which is more verbose and allows the user to specifically keep defaults, or override them.

The most safe way to have everything intact, including the API setup, would be:

icinga2 api setup
icinga2 pki save-cert
icinga2 node setup
<custom config foo>

Likely our colleagues can add more insights here on how they've automated and scripted their setups. @mkayontour @lbetz @widhalmt @lazyfrosch @dgoetz @bobapple

Maybe it would also be a thing to just use puppet and puppet-icinga2 to run the client setup with apply ... then everything from installation to the final configuration is handled with a tested source.

Anyhow, I think that PS knows all the best practices for such a thing, and I am going back to syncing some zone trees.

Cheers, Michael

Thomas-Gelf commented 6 years ago

Hi @dnsmichi,

thanks for your valuable input. So, first of all what I read is that those paths are still working, that's what was my understanding, that's why they are still there. We all agree that we could do better for 2.8, so first we need to detect the version. I'll assume that icinga2 is always on your PATH and run:

icinga2 --version | head -n1 \
 | sed -r 's/.+version: [rv]([0-9]+\.[0-9]+).+/\1/'

awk might be simpler, but apart from that: [Q1] is this a safe way to determine the Icinga 2 major/minor version, can I assume that this doesn't break?

Given the version we could continue to deploy 2.6/2.7 agents like we're doing it now, from 2.8 on we do not care about the ssl path and let Icinga handle it. We still need to make some assumption on the path itself, as it is vital to check whether a certificate exists or not. [Q2] Is there a way to ask the binary for the SSL directory it is going to use?

Then, regarding the Puppet module:

As far as I know the Puppet module didn't fix the ssl path issue, your suggestion in the unresolved #398 points to the strategy used in this script: do nothing for now
for the Binary the Puppet-Module uses absolute paths and hard-codes them for every distribution, there have already been related issues
Puppet users will not use this script, they have Puppet. On the other side, you'll not be able to convince people not using Puppet to introduce it just to install a monitoring agent. This script is there for convenience for those who have no other automation in place.

And last, you eventually misunderstood the note regarding the running daemon. After installation, icinga2 might not be running. It might never have been. It usually immediately runs on Debian systems and it doesn't on RedHat systems. In case it has never been started, as far as I remember there used to be missing directories. If I recall it correctly the pki tools used to fail without them, that's why the script runs prepare-dirs. [Q3] Can I assume that icinga2 pki runs flawlessly on 2.8 when prepare-dirs has never been triggered before?

Your suggestions then point to tooling that writes configuration files. There are many minor and major issues with that approach, I'd prefer not going into details here. The existing script runs the same commands as the Puppet module, api/node setup do not fit for this use case.

But that's not a big problem, I think we can get this going with answers to [Q1-3].

Thanks a lot! Thomas

dnsmichi commented 6 years ago

I'll come back to you once my list of PR reviews and IcingaDB issues are cleared up. Likely in CW34 or CW36.

Cheers, Michael

lazyfrosch commented 6 years ago

Should be fixed with #1585

Please test if you like

Icinga / icingaweb2-module-director