search

mirceaulinic / salt-sproxy

Salt plugin to automate the management and configuration of (network) devices at scale, without running (Proxy) Minions.
https://salt-sproxy.readthedocs.io/en/latest/
Apache License 2.0
117 stars 19 forks source link

salt-sproxy fails to parse ansible-roster return #162

Open felskrone opened 4 years ago

felskrone commented 4 years ago

Describe the bug salt-sproxy fails to evaluate targets from ansible roster

Steps To Reproduce The volumes/config_exchange contains my configs and pillars and get mounted into the container at /mnt. They are externally generated from another container into a shared volume.

docker pull mirceaulinic/salt-sproxy:allinone-2020.7.0
docker run -it -v /var/lib/docker/volumes/config_exchange/_data/salt_napalm:/mnt mirceaulinic/salt-sproxy:allinone-2020.7.0 bash

In the container i simply symlink the configs into place, removing the existing master and roster

cd /etc/salt
rm -vf master roster
ln -s /mnt/etc/salt/master .
ln -s /mnt/etc/salt/roster .

cd /srv
ln -s /mnt/srv/pillar .
ln -s /mnt/srv/salt .

These are all currently empty /srv/salt/ /srv/salt/ext /srv/pillar/ext

This is what the environment looks like when executing sproxy commands:

root@b3528bf2e5f8:/etc/salt# cat /etc/salt/master

pillar_roots:
  base:
    - /srv/pillar
    - /srv/pillar/ext

file_roots:
  base:
    - /srv/salt/
    - /srv/salt/ext

proxy_roster: ansible
roster_file: /etc/salt/roster

root@b3528bf2e5f8:/etc/salt# cat /etc/salt/roster
all:
  hosts:
    10.220.0.11:
    10.220.0.12:

The inventory seems to work fine

root@b3528bf2e5f8:/etc/salt# ansible-inventory -i /etc/salt/roster --list
{
    "_meta": {
        "hostvars": {
            "10.220.0.11": {},
            "10.220.0.12": {}
        }
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    },
    "ungrouped": {
        "hosts": [
            "10.220.0.11",
            "10.220.0.12"
        ]
    }
}

Trying to get a preview of the targets

root@b3528bf2e5f8:/etc/salt# salt-sproxy -l debug \* --preview-target
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Syncing grains
[DEBUG   ] LazyLoaded saltutil.sync_grains
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly set in /etc/salt/minion
[DEBUG   ] Found minion id from generate_minion_id(): b3528bf2e5f8
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Unable to derive osmajorrelease from osrelease_info '('proxy',)'. The osmajorrelease grain will not be set.
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[WARNING ] /usr/local/lib/python3.6/site-packages/salt/transport/ipc.py:292: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  self.unpacker = msgpack.Unpacker(encoding=encoding)

[INFO    ] Creating module dir '/var/cache/salt/master/extmods/grains'
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating roots fileserver cache
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base)
[INFO    ] Caching directory '_grains/' for environment 'base'
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_grains'
[DEBUG   ] LazyLoaded local_cache.prep_jid
[INFO    ] Runner completed: 20200807113632764881
[DEBUG   ] []
[DEBUG   ] Syncing modules
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[INFO    ] Creating module dir '/var/cache/salt/master/extmods/modules'
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)
[INFO    ] Caching directory '_modules/' for environment 'base'
[DEBUG   ] Returning file list from cache: age=3 cache_time=20 /var/cache/salt/master/file_lists/roots/base.p
[WARNING ] /usr/local/lib/python3.6/site-packages/salt/payload.py:149: DeprecationWarning: encoding is deprecated, Use raw=False instead.
  ret = msgpack.loads(msg, use_list=True, ext_hook=ext_type_decoder, encoding=encoding)

[DEBUG   ] In saltenv 'base', looking at rel_path '_modules/netbox.py' to resolve 'salt://_modules/netbox.py'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/master/files/base/_modules/netbox.py' to resolve 'salt://_modules/netbox.py'
[DEBUG   ] Local cache dir: '/var/cache/salt/master/files/base/_modules'
[INFO    ] Runner completed: 20200807113634956892
[DEBUG   ] []
[DEBUG   ] LazyLoaded proxy.execute
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Guessing ID. The id can be explicitly set in /etc/salt/minion
[DEBUG   ] Found minion id from generate_minion_id(): b3528bf2e5f8
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Unable to derive osmajorrelease from osrelease_info '('proxy',)'. The osmajorrelease grain will not be set.
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Computing the target using the ansible Roster
[DEBUG   ] LazyLoaded ansible.targets
[DEBUG   ] Called salt.cmd runner with minion function cmd.run
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Unable to derive osmajorrelease from osrelease_info '('proxy',)'. The osmajorrelease grain will not be set.
[DEBUG   ] LazyLoaded cmd.run
[INFO    ] Executing command 'ansible-inventory -i /etc/salt/roster --list' in directory '/root'
[DEBUG   ] stdout: {
    "_meta": {
        "hostvars": {
            "10.220.0.11": {},
            "10.220.0.12": {}
        }
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    },
    "ungrouped": {
        "hosts": [
            "10.220.0.11",
            "10.220.0.12"
        ]
    }
}
[DEBUG   ] output: {
    "_meta": {
        "hostvars": {
            "10.220.0.11": {},
            "10.220.0.12": {}
        }
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    },
    "ungrouped": {
        "hosts": [
            "10.220.0.11",
            "10.220.0.12"
        ]
    }
}
[DEBUG   ] LazyLoaded local_cache.prep_jid
[DEBUG   ] LazyLoaded nested.output
Exception occurred in runner proxy.execute: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/salt/client/mixins.py", line 377, in low
    data['return'] = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/salt_sproxy/_runners/proxy.py", line 1458, in execute
    rtargets_roster = roster_modules[roster](_tgt, tgt_type=_tgt_type)
  File "/usr/local/lib/python3.6/site-packages/salt/roster/ansible.py", line 121, in targets
    __context__['inventory'] = __utils__['json.loads'](__utils__['stringutils.to_str'](inventory))
  File "/usr/local/lib/python3.6/site-packages/salt/utils/context.py", line 236, in __getitem__
    return self._dict()[key]
KeyError: 'json.loads'
[INFO    ] Runner completed: 20200807113637045024
[DEBUG   ] Runner return: Exception occurred in runner proxy.execute: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/salt/client/mixins.py", line 377, in low
    data['return'] = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/salt_sproxy/_runners/proxy.py", line 1458, in execute
    rtargets_roster = roster_modules[roster](_tgt, tgt_type=_tgt_type)
  File "/usr/local/lib/python3.6/site-packages/salt/roster/ansible.py", line 121, in targets
    __context__['inventory'] = __utils__['json.loads'](__utils__['stringutils.to_str'](inventory))
  File "/usr/local/lib/python3.6/site-packages/salt/utils/context.py", line 236, in __getitem__
    return self._dict()[key]
KeyError: 'json.loads'

[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Closing IPCMessageClient instance

Expected behavior A return of all minions / targets defined in roster

$ salt-sproxy \* --preview-target
- 10.220.0.11
- 10.220.0.12

Versions Report Print the output from salt-sproxy -V inside the backticks below:

Salt Version:
           Salt: 2019.2.0
    Salt SProxy: 2020.7.0

Dependency Versions:
        Ansible: 2.8.1
           cffi: 1.14.0
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.11.2
     junos-eznc: 0+unknown
       jxmlease: 1.0.1
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
         NAPALM: 3.1.0
       ncclient: 0.6.7
        Netmiko: 3.1.1
       paramiko: 2.7.1
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pyeapi: 0.8.3
         pygit2: Not Installed
       PyNetBox: 4.0.6
          PyNSO: Not Installed
         Python: 3.6.11 (default, Jun 30 2020, 19:44:32)
   python-gnupg: Not Installed
         PyYAML: 5.3.1
          PyZMQ: 19.0.1
            scp: 0.13.2
          smmap: Not Installed
        textfsm: 1.1.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: debian 9.12
         locale: UTF-8
        machine: x86_64
        release: 4.19.0-9-amd64
         system: Linux
        version: debian 9.12

Additional context i tried the -dev version "mirceaulinic/salt-sproxy:allinone-dev" as well, same problem.

If i can be of any more help, please let me know.

felskrone commented 4 years ago

I got it to work with

roster: file
10.220.0.11:
10.220.0.12:

which is sufficient for me because i just need a flat file of minion_ids i can target and im not going to use salt-ssh in the future.

mirceaulinic commented 4 years ago

I'm seeing some suspicious behaviour, reopening to investigate it soon. Thanks for reporting @felskrone!