[BUG] salt-minion uses reverse dns to generate minion_id instead of local hostname/fqdn

[rossengeorgiev]

Description salt-minion uses reverse dns to generate minion_id instead of local hostname/fqdn

• 2 x VMs (Vmware), CentOS 7

Steps to Reproduce the behavior

1. two servers, salt-master and minion
2. minion is a server with ip, which has a reverse DNS record different from the server configured hostname/fqdn
3. start salt-minion and look at /etc/salt/minion_id

Expected behavior /etc/salt/minion_id to contain the server local hostname/fqdn, not whatever reverse DNS query returns for the IP address

Screenshots

[root@host1 ~]# hostname -f
host1.example.com
[root@host1 ~]# rm /etc/salt/minion_id
rm: remove regular file ‘/etc/salt/minion_id’? y
[root@host1 ~]# systemctl restart salt-minion
[root@host1 ~]# cat /etc/salt/minion_id
another-name-from-dns.example.com
[root@host1 ~]#
[root@host1 ~]# host host1.example.com
host1.example.com has address 10.10.10.5
[root@host1 ~]# host 10.10.10.5
5.10.10.10.in-addr.arpa domain name pointer another-name-from-dns.example.com.
[root@host1 ~]# rpm -q salt
salt-3002.2-1.el7.noarch
[root@host1 ~]# cat /etc/hostname
host1.example.com

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3002.2 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.11.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Nov 16 2020, 16:55:22) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: centos 7 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1160.66.1.el7.x86_64 system: Linux version: CentOS Linux 7 Core ```

[Jibun-no-Kage]

Triage? I suggest the above is specific, and clear. Can you not recreate the behavior with ease? I just tested this and found it is consistent, as in UNEXPLICITED behavior.

For example, if the given target system has multiple fqdn entries in DNS, the result is one of the DNS entries but not consistently the correct one, or the defined interface, is used. Even if you force the minion to a specific interface, so that it has to use a specific IP address, the result is an inconsistent fqdn maybe returned, and thus even the key assignment maybe the wrong name for the assigned interface.

Minion: 192.168.1.1 Ethernet interface eth0 DNS bozo-ethernet.dd.org 192.168.1.2 Wifi interface wlan0 bozo-wireless.dd.org FQDN/hostname defined in the OS bozo.dd.org

Why is bozo-wireless.org EVER returned to the master for key acceptance? It is not the OS hostname, it is not even the DNS bame bound to the IP for the defined minion interface, i.e. eth0.

Even if I force the minion to use eth0, the actual name the master receives for the key acceptance CAN BE bozo-wireless.dd.org.

I strongly suggest, this is NOT a good behavior, especially when the minion configuration explicitly defines interface is bound to eth0, and as the OP noted, the expected behavior is to use the ACTUAL hostname/fqdn defined in the minion OS, I would say unless an explicit interface is defined.

@rossengeorgiev. as a work around... change the /etc/salt/minion_id file to the name you want used. Which must resolve via DNS of course. I setup at small script that checks the minion_id file, and updates it, and restarts the minion service. Of course, you have to accept the key at the master(s) applicable, but at least the correct name is in the master key list.

[rossengeorgiev]

This issue shows up in a whole bunch of cases, and is another annoying thing you have to deal with when using salt. salt-ssh has/had the same issue, where you would target a hostname, but salt-ssh would resolve the ip, and then resolve the rDNS for that ip, and then connect to the hostname from the rDNS. Had instances where that would be a completely different box. Granted, the DNS records should be fixed, but this is still absolutely insane and stupid behaviour to have. Nothing else does that.

A recent example, of this specific issue with Ubuntu on Openstack. For context, openstack can MITM DNS requests and insert responses, such as PTR for the server that is equal to the server name. Naturally, salt-minion v3004, instead of using the configured FQDN on Ubuntu (/etc/hostname), it does rDNS and uses that. Just plain stupid. Double stupid, since on Ubuntu, when you install salt-minion the service is started immediately, before you get a chance to update the config. Another stupid behaviour. So now, you have to stop the service, update minion config, update minion_id, remove minions pki keys and re-start it.

Oh wait, you have to remove pki keys? YES, because the master thinks the new minion is the previous minion with the old minion_id using the PKI. Another stupid behaviour. Using salt sometimes feels like someone has spilled LEGO bricks all over the floor when you just are just trying to walk across the room.

[Jibun-no-Kage]

LOL... LEGO reference.

I stopped using Ansible because it was such a PITA to do anything, and the playbooks were painful to read, never mind try to maintain over time, SaltStack is SO much easier to develop state files for, far easier to learn and train for. But it has these really stupid quirks, that cripple it.

This host name issue, is one such. Oh, and there are times when the DNS is actually CORRECT, because it needs to be a bit odd for other reasons, such as CNAME aliasing, and multi-homing of IP addresses, which can be required. So just falling back to DNS needs to be fixed, is not realistic at times, under various conditions or situations. SaltStack seems to only be functional when the forward and reverse domains are simple one-to-one design. Well, I have many-to-one forward, and one-to-one reverse. This is by design in our specific use case. But with SaltStack we have had to use static host name assignment (via the minion_id file).

Another quirk issue is the lack of process tracking during a long executing state. Don't even suggest events. Don't even suggest parsing the logs after completion. That is not what is needed, or wanted, what is wanted is a real time PROGRESS indicator. I have seen this requested over an over, and SaltStack just refused to address the issue, the reasons vary, and still people ask for it, because it is wanted and needed.

I guess that is a bit of a rant, but I am sure the frustration out in the user population is there, some of us are just more vocal, cough, take the time to text it.

[rossengeorgiev]

For me Ansible is sane, not without problems, but definitely had better experience. Saltstack has good concept, but it needs to do the hard thing and start over clean, with well defined concept. Ansible has a lot more structure, and doesn't try to do everything in one package, instead has a repository for collections. And those can be updated very quickly, a recent change took me a day, while on the salt repo, someone might look at an issue or PR, maybe some time after the first time stalebot closes it. With salt, I've end up having duplicate of modules with the fixes in the gitrepo, so I get on with my work in a reasonable timeframe.

Anyway, I won't make any more off-topic comments. Not the right place to vent frustration.

[Jibun-no-Kage]

Interesting, that your Ansible experience is opposite of mine, at a given level. I just found SaltStack was had an easier learning curve, for the environment I was in. Different people, different challenges. I don't disagree that SaltStack could use a strategic review, I think that is true. I think every product/solution, as it ages/matures needs some type of sanity check as part of its routine existence. I am sure my experience with Ansible is not done, of course. :)

[dmurphy18]

@rossengeorgiev The version of that you stated is 3002.2 which is long EOL, can you try with the latest version of Salt which is 3006.1 and see if the problem still occurs.

Note: in the upcoming Salt 3007 the default for _enable_fqdnsgrains has been changed from True to False, however it is still True in 3006.1.

Note that the reverse DNS query will return the IP address as configured for whatever the machine that the salt-minion is residing on, which is outside of the salt-minion control.

For example: socket.getfqdn(socket.gethostbyaddr(ip)[0]) where ip is obtained from the network interfaces

[dmurphy18]

Closing due to unresponsive, please re-open if further information to consider concerning this issue