Core napalm grains assigned after custom grains

[jtraub91]

Description of Issue

According to https://docs.saltstack.com/en/latest/topics/grains/#precedence core grains are assigned first and thus should be overwritten by custom grains in _grains/. However, this is not the case for core grains included in salt/grains/napalm.py

Setup

Need an actual or virtual device capable of being controlled by napalm proxy minion. Need to then configure proxy nominally via pillar data.

Steps to Reproduce Issue

Create custom grain 'interfaces' to override salt-napalm included 'interfaces'. Does not work as described in link above

Test to demonstrate problem (some output modified / redacted) Created custom grain in issue.py

# /srv/salt/_grains/issue.py

import logging

__proxyenabled__ = ['napalm']

logger = logging.getLogger(__name__)

def foo(grains):
    logger.debug("foo")
    logger.debug(grains)
    return {'interfaces': 'test_value'}

but after syncing:

# salt <proxy-minion-id> grains.item interfaces
<proxy-minion-id>:
    interfaces:
        - Loopback
        ...

and in logs from /var/log/salt/proxy

2019-09-19 15:53:27,624 [salt.loaded.ext.grains.issue                                  :8   ][DEBUG   ][23652] foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.issue                                  :9   ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': '4.3.2', u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', u'mem_total': 0}
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :10  ][DEBUG   ][23652] core_foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :11  ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': '4.3.2', u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', 'interfaces': 'test_value', u'mem_total': 0}
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :10  ][DEBUG   ][23652] foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :11  ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': 2, u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'id': 'ios_xr_1', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', 'interfaces': 'test_value', u'mem_total': 0}

'interfaces' initially didn't exist but then was added the custom grain value of 'test_value' (which must have been subsequently overwritten as seen by output of salt <proxy-minion-id> grains.item interfaces command above)

Furthermore later in /var/log/salt/proxy we see

2019-09-19 15:53:27,827 [netmiko                                                       :463 ][DEBUG   ][23652] read_channel:
2019-09-19 15:53:27,828 [netmiko                                                       :385 ][DEBUG   ][23652] write_channel: <?xml version="1.0" encoding="UTF-8"?><Request MajorVersion="1" MinorVersion="0"><Get><Operational><SystemTime/><PlatformInventory/>        </Operational></Get></Request>
2019-09-19 15:53:31,434 [netmiko                                                       :463 ][DEBUG   ][23652] read_channel: <?xml version="1.0" encoding="UTF-8"?>
<Response MajorVersion="1" MinorVersion="0"><Get><Operational><SystemTime MajorVersion="3" MinorVersion="0"><Clock><Year>2019</Year><Month>9</Month><Day>19</Day><Hour>15</Hour><Minute>45</Minute><Second>11</Second><Millisecond>247</Millisecond><Wday>4</Wday><TimeZone>UTC</TimeZone>
...
</Get><ResultSummary ErrorCount="0"/></Response>
XML>

Showing the xml being requested and responded to (likely from the core salt-napalm grains)

This isn't the behavior of the more 'standard' core grains:

# /srv/salt/_grains/test_grains.py
__proxyenabled__ = ['napalm']

def core_foo(grains):
     return {'zmqversion': 2}

after syncing:

# salt <proxy-minion-id> grains.item zmqversion
<proxy-minion-d>:
    zmqversion:
        2

Versions Report

bash-4.4$ salt --versions-report Salt Version: Salt: 2019.2.0

Dependency Versions: cffi: 1.12.3 cherrypy: unknown dateutil: 2.7.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.9.6 libgit2: 0.27.7 libnacl: Not Installed M2Crypto: 0.33.0 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: 2.17 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: 0.27.4 Python: 2.7.16 (default, May 6 2019, 19:35:26) python-gnupg: 0.4.2 PyYAML: 5.1.1 PyZMQ: 18.1.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 5.1.1 ZMQ: 4.3.2

System Versions: dist: locale: UTF-8 machine: x86_64 release: 3.10.0-862.3.2.el7.x86_64 system: Linux version: Not Installed

[jtraub91]

PR opened here https://github.com/saltstack/salt/pull/54853

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[sagetherage]

@Akm0d you did triage on this ticket, is it a regression? what is the criticality?