search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.03k stars 5.47k forks source link

salt-master immediately denies minion keys #66162

Open snarl817 opened 5 months ago

snarl817 commented 5 months ago

Description The salt-master denied minion keys for several minions. I deleted the keys and restarted the minon. On the next connect attempt, the minion reported that the master rejected the key. The log on the master reports that the key in pending did not match. I have REPEATEDLY deleted and recreated the key, but the master always denies it with the same error that it doesn't match. It shows that the key is BOTH denied AND accepted.

Setup I upgraded from Fedora 37 to Fedora 39. After the upgrade, the master gave an error applying states, and that's when I discovered that Fedora 39 isn't supported. I created a backup of my master, pilar, and /etc/salt directory structures and installed a NEW VM using Fedora 38. The backup was restored, and saltstack was installed using the bootstrap script.

Please be as specific as possible and give set-up details.

LOG from the minion: 2024-02-29 11:43:20,798 [salt.crypt :835 ][CRITICAL][18259] The Salt Master has rejected this minion's public key! To repair this issue, delete the public key for this minion on the Salt Master and restart this minion. Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

LOG from the master: 2024-02-29 11:43:20,759 [salt.channel.server:484 ][ERROR ][9337] Authentication attempt from f1o-dns failed, the public key in pending did not match. This may be an attempt to compromise the Salt cluster.

md5 hash of the minion key on the master: b4258898b334b03809f44435c165aae8 f1o-dns

md5 hash of the minion's public key on the minion: b4258898b334b03809f44435c165aae8 minion.pub

Steps to Reproduce the behavior Bootstrap install salt master and minion on a fresh Fedora 38 system. Start the minion. Accept the key. Check the logs.

Expected behavior The master should NOT deny the key as it matches.

Screenshots Accepted Keys: f1o-dns: a5:04:eb:4f:43:66:18:8a:8e:0e:35:a3:98:94:59:2f:c5:ba:fd:94:0e:0f:0b:1e:01:b0:da:87:bc:eb:1b:25 Denied Keys: f1o-dns: a5:04:eb:4f:43:66:18:8a:8e:0e:35:a3:98:94:59:2f:c5:ba:fd:94:0e:0f:0b:1e:01:b0:da:87:bc:eb:1b:25

Versions Report

salt --versions-report MASTER: Salt Version: Salt: 3006.7 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: 3.20.0 pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.15.1 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: fedora 38 locale: utf-8 machine: x86_64 release: 6.7.6-100.fc38.x86_64 system: Linux version: Fedora Linux 38 MINION: salt-3005.1-1.el7.noarch salt-minion-3005.1-1.el7.noarch ```yaml N/A ```

Additional context I have secrets in the salt-pillar, but salt can't decrypt them...I need to check my gpg setup, but I assume this is an artifact of the migration.

welcome[bot] commented 5 months ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

snarl817 commented 5 months ago

This can be closed, I found the issue...interestingly enough, as I was trying to fix the gpg problem. The package upgrade REPLACED MY MASTER CONFIG with a default. I dutifully backed it up and restored it on the new VM. I didn't think about the fact that the base_dir and piller_dir being reset to defaults meant that it ALSO set the user to "salt" instead of root. As a result, the master couldn't ACTUALLY read any of the keys, and rejected them.

After changing the user to root, deleting the denied keys and restarting the minions, I'm able to accept the minion keys.

Lesson learned: backup the config file BEFORE any system upgrades are performed. Also, snapshot the VM storage, too.

snarl817 commented 4 months ago

...turns out I spoke too soon. The master once again denied the minion's key. salt-key -f shows that the minion's key is both ACCEPTED and DENIED. When I accept the key from the master, it prints the minion id twice: Key for minion f1o-dns accepted. Key for minion f1o-dns accepted.

Same when I subsequently delete the key. The log message remains the same: Authentication attempt from f1o-dns failed, the public key in pending did not match. This may be an attempt to compromise the Salt cluster.

There is no further information about WHY it thinks the key is different: and md5 hash of the key on the master and the key on the minion still indicates that the files are identical.

whytewolf commented 4 months ago

so the second problem is a duplicate of https://github.com/saltstack/salt/issues/66126 and it looks like the first problem is fixed.

the second problem is already being addressed in https://github.com/saltstack/salt/pull/66153 if you want to test that change and see if it works.

Selected6280 commented 4 months ago

I just updated our master to 3006.7 and minion on Centos 7 with version 3005.x are now all denied. Ubuntu doesn't seem to have this problem. I delete the keys from one minion on the master. But it come backs twice. Once in unaccepted and once in denied. The keys are the same. diff /etc/salt/pki/master/minions_pre/server /etc/salt/pki/master/minions_denied/server doesn't show a difference.

whytewolf commented 4 months ago

I just updated our master to 3006.7 and minion on Centos 7 with version 3005.x are now all denied. Ubuntu doesn't seem to have this problem. I delete the keys from one minion on the master. But it come backs twice. Once in unaccepted and once in denied. The keys are the same. diff /etc/salt/pki/master/minions_pre/server /etc/salt/pki/master/minions_denied/server doesn't show a difference.

there wouldn't be an expected difference. see https://github.com/saltstack/salt/issues/66126

the issue is the newline is being scrubbed in one place but not all.

Selected6280 commented 4 months ago

Is this then fixed in the next master 3006.8?

whytewolf commented 4 months ago

Is this then fixed in the next master 3006.8?

that should be the case yes.

jeff350 commented 1 month ago

I have seen this occurring 3006.8 masters with some older 3004 and 3005 minions working and others having their keys denied as described above.

jmozd commented 3 weeks ago

Seems the updated fix (covering also older minion versions) is in v3007.1, see https://github.com/saltstack/salt/commit/b7f44da84926abd95f95028e54cd6f5844d86ebb