search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.13k stars 5.47k forks source link

target grains ip_interface no return received #49626

Closed PabloLemos closed 6 years ago

PabloLemos commented 6 years ago

Description of Issue/Question

I need to list those machines that are located in a certain network. For this I usually use the grains ip_interfaces and so far it has worked for me; but I'm currently getting the following error when I try to list the machines.

[master]$ sudo salt -C 'G@ip_interfaces:*:10.0.1.*' test.version
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received

Filtering the logs I see the following error message

2018-09-12 14:09:01,744 [salt.utils.minions                                    :696 ][ERROR   ][5648] Failed matching available minions with compound pattern: G@ip_interfaces:*:10.0.1.*
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 683, in check_minions
    _res = check_func(expr, delimiter, greedy)
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 578, in _check_compound_minions
    _results = engine(*engine_args)
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 325, in _check_grain_minions
    return self._check_cache_minions(expr, delimiter, greedy, 'grains')
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 315, in _check_cache_minions
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 579, in subdict_match
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 549, in _dict_match
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 526, in _match
    return fnmatch.fnmatch(six.text_type(target).lower(), pattern.lower())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

However, when I consult for other grains; for example kernel, I get the machine listing

[master]$ sudo salt -C 'G@kernel:Linux' test.version
minion1:
    2014.1.4
minion2:
    2018.3.2
minion3:
    2018.3.2
minion4:
    2017.7.0
minion5:
    2017.7.0

Setup

Steps to Reproduce Issue

[master]$ sudo salt -C 'G@ip_interfaces:*:10.0.1.*' test.version -l trace
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-cli.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-cli.conf
[DEBUG   ] Changed git to gitfs in master opts' fileserver_backend list
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: gadvirtu.srv.gva.es
[DEBUG   ] Missing configuration file: /root/.saltrc
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-cli.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-cli.conf
[DEBUG   ] Changed git to gitfs in master opts' fileserver_backend list
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None)
[DEBUG   ] Popen(['git', 'version'], cwd=/root, universal_newlines=False, shell=None)
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'master_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG   ] Trying to connect to: tcp://127.0.0.1:4506
[TRACE   ] Inserted key into loop_instance_map id 140063542022512 for key (u'/etc/salt/pki/master', u'master_master', u'tcp://127.0.0.1:4506', u'clear') and process 31276
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc
No minions matched the target. No command was sent, no jid was assigned.
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {}
ERROR: No return received
[master]$ tail -100 /var/log/salt/master
2018-09-12 14:17:56,727 [salt.utils.minions                                    :696 ][ERROR   ][5648] Failed matching available minions with compound pattern: G@ip_interfaces:*:10.0.1.*
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 683, in check_minions
    _res = check_func(expr, delimiter, greedy)
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 578, in _check_compound_minions
    _results = engine(*engine_args)
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 325, in _check_grain_minions
    return self._check_cache_minions(expr, delimiter, greedy, 'grains')
  File "/usr/lib/python2.7/site-packages/salt/utils/minions.py", line 315, in _check_cache_minions
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 579, in subdict_match
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 549, in _dict_match
    exact_match=exact_match):
  File "/usr/lib/python2.7/site-packages/salt/utils/data.py", line 526, in _match
    return fnmatch.fnmatch(six.text_type(target).lower(), pattern.lower())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

Versions Report

[master]$ salt --versions-report
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.4
      gitpython: 2.1.11
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jul 12 2017, 17:32:34)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 2.0.4
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.9 Santiago
         locale: UTF-8
        machine: x86_64
        release: 2.6.32-696.6.3.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.9 Santiago

Thanks a lot

Ch3LL commented 6 years ago

i'm having a hard time replicating this:

➜  salt git:(e7893aedcf) sudo salt -C 'G@ip_interfaces:*:10.62.147.*' test.version
the_cake_is_a_lie:
    2018.3.2

Did you recently upgrade? i'm also wondering if there is something in the minion cache from teh older minion versions that is possibly causing the issue

PabloLemos commented 6 years ago

@Ch3LL Thanks for answering so fast!!!

Yes, im recently upgrade the master.

My salt infrastructure is extensive (about 800 minions). I have filtered by the salt version to go over the grains ip_interfaces and I have not obtained any error information in the logs.

An example of 2014 versions:

[master] $ salt -C 'G@saltversion:2014. *' grains.item ip_interfaces saltversion
minion1:
    ----------
    ip_interfaces:
        ----------
        eth0:
            - 10.178.247.50
        eth1:
            - 10.179.246.50
        the:
            - 127.0.0.1
    saltversion:
        2014.1.4
minion2:
    ----------
    ip_interfaces:
        ----------
        lo0:
            - 127.0.0.1
        vnet0:
            - 10.27.209.61
        vnet407001:
            - 0.0.0.0
        vnet421001:
            - 11.27.162.142
        vnet601001:
            - 0.0.0.0
    saltversion:
        2014.1.4

An example of 2015 versions:

[master]$ salt -C 'G@saltversion:2015.*' grains.item ip_interfaces saltversion
minion1:
    ----------
    ip_interfaces:
        ----------
        eth0:
            - 10.27.208.252
        eth1:
            - 11.27.246.252
        lo:
            - 127.0.0.1
            - 127.0.0.2
    saltversion:
        2015.8.1

minion2:
    ----------
    ip_interfaces:
        ----------
        eth0:
            - 10.27.170.41
            - fe80::250:56ff:fe89:6177
        lo:
            - 127.0.0.1
            - ::1
    saltversion:
        2015.8.1

minion3:
    ----------
    ip_interfaces:
        ----------
        e1000g2:
            - 10.27.164.133
        e1000g3:
            - 10.27.245.254
        lo0:
            - 127.0.0.1
    saltversion:
        2015.8.7

I have also tried to empty the cache of all the minions by removing the directory:

rm -rf /var/cache/salt/master/minions/*

and forcing the update using the command:

salt '*' saltutil.sync_grains refresh=True

Can I perform some command to get a more accurate debug of the targetting functions?

Thanks a lot!!

PabloLemos commented 6 years ago

Checking the information of some windows servers I have verified that the name of the network interface has extended ASCII characters:

windows_minion_1:
    ----------
    ip_interfaces:
        ----------
        Intel(R) 82574L Gigabit Network Connection:
            - 10.11.10.1
windows_minion_2:
    ----------
    ip_interfaces:
        ----------
        Conexión de red Intel(R) PRO/1000 MT #2:
            - 10.11.10.2
windows_minion_3:
    ----------
    ip_interfaces:
        ----------
        vmxnet3 Ethernet Adapter:
            - 10.11.10.3
windows_minion_4:
    ----------
    ip_interfaces:
        ----------
        Citrix Virtual Adapter - Deterministic Network Enhancer Miniport:
            - 0.0.0.0
        vmxnet3 Ethernet Adapter - Deterministic Network Enhancer Miniport:
            - 10.11.10.4

Can this be the cause of the exception that appears in the salt logs?

Thanks!!

Ch3LL commented 6 years ago

thanks for tracking that down. i believe thats exactly the issue.

I was able to replicate by hacking the code with:

diff --git a/salt/grains/core.py b/salt/grains/core.py
index 8545d4368c..5ba212b21e 100644
--- a/salt/grains/core.py
+++ b/salt/grains/core.py
@@ -1946,6 +1946,7 @@ def ip_interfaces():
             if 'address' in secondary:
                 iface_ips.append(secondary['address'])
         ret[face] = iface_ips
+    ret['Conexión de red Intel(R) PRO/1000 MT #2'] = ['1.1.1.1']
     return {'ip_interfaces': ret}

ping @terminalmage any ideas here?

terminalmage commented 6 years ago

The matching system was not intended to support multiple wildcards like this. The wildcard only works when it's in the value being matched, not in any of the intermediate nested levels of keys. To make it work for intermediate nested levels would require some significant restructuring, which would need to be done in a future feature release.

I have a question though: Were you aware that there is a subnet matcher? This would seem to be a much easier way of doing what you want:

salt -S 10.0.1.0/24 test.version
terminalmage commented 6 years ago

Hmm it seems that support for intermediate wildcards may have been added since I last looked at this code, I'm investigating. But still, I think that the subnet matcher is a better way of doing what you want.

PabloLemos commented 6 years ago

Hi @terminalmage

The subnet matcher I understand is for the service IPs of the agent, right? my agents have different network interfaces: the service interface and a couple of non-routed network interfaces (NFS, etc).

Using network identification can I interrogate my agents for those interfaces that are NOT routed?

terminalmage commented 6 years ago

Yes, it should still work. Did you try it?

The ip_interfaces and ip6_interfaces grains contain the same contents as the ipv4 and ipv6 grains, which are used in the the subnet matcher. The one difference is that the ipv4 and ipv6 grains are just lists of IP addresses, while ip_interfaces and ip6_interfaces are dictionaries mapping interface names to lists of IP addresses.

terminalmage commented 6 years ago

AH! I found the problem. So yes, wildcard matching was indeed added at some point, but while I was in the process of writing new unit tests for this I discovered a completely unrelated bug! If the data being matched on the level after the wildcard is the value side of a key/value pair in a dict, the match fails. However, when it's a list (as in your example) the match succeeds.

I'm working on a fix for that right now, as well as for the UnicodeDecodeError.

I still recommend the subnet matcher though :wink:

terminalmage commented 6 years ago

See https://github.com/saltstack/salt/pull/49791

PabloLemos commented 6 years ago

Thanks a lot!!

doesitblend commented 5 years ago

ZD-3024