under salt-ssh, saltutil.refresh_pillar is allowed to run, but always returns False

[shallot]

Description of Issue/Question

For some reason, with salt-ssh every call to saltutil.refresh_pillar returns False, and I've no idea how to diagnose what's wrong. Is there anything wrong? All of the files do seem to be deployed to the minion.

Adding -l debug to the command line doesn't appear to help - there's a lot of prologue and epilogue added, but no apparent clarification on the core False.

Setup

% cat salt/roster
minion-name:
  host: 1.2.3.4
  user: ubuntu
  priv: /opt/key-josip-2017-04-05
  sudo: True
  thin_dir: /.salt-ssh-thin-dir
  port: 1234

% cat salt/master
root_dir: /opt/stuff/salt
log_file: var/log/salt/salt.log
ssh_log_file: /opt/stuff/salt/var/log/salt/ssh
file_roots:
  base:
    - /opt/stuff/salt/states
ext_pillar:
  - file_tree:
      root_dir: /opt/stuff/salt/ext_pillar
      follow_dir_links: False
      keep_newline: True

% find salt/ext_pillar/ -ls
  2739    4 drwxr-xr-x   3 josip    josip        4096 Mar 17 09:41 salt/ext_pillar/
 10288    4 drwxr-xr-x   3 josip    josip        4096 Apr  5 22:38 salt/ext_pillar/hosts
 14517    4 drwxr-xr-x   7 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name
 14567    4 -rw-r--r--   1 josip    josip         645 Mar 23 10:16 salt/ext_pillar/hosts/minion-name/local-debug
 14548    4 drwxr-xr-x   2 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/pgbouncer
 14549    4 -rw-r--r--   1 josip    josip         148 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/pgbouncer/userlist.txt
 14559    4 drwxr-xr-x   2 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/apache2
 14560    4 -rw-r--r--   1 josip    josip        1908 Mar 22 14:30 salt/ext_pillar/hosts/minion-name/apache2/local-stuff.conf
 14538    4 drwxr-xr-x   3 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/userdata
 14539    4 drwxr-xr-x   2 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/userdata/stuff
 14547    4 -rw-r--r--   1 josip    josip          89 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/userdata/stuff/pgpass
 14546    4 -rw-r--r--   1 josip    josip         398 Apr 28  2015 salt/ext_pillar/hosts/minion-name/userdata/stuff/authorized_keys
 14544    4 -rw-------   1 josip    josip        1675 Mar 17 11:27 salt/ext_pillar/hosts/minion-name/userdata/stuff/id_rsa
 14545    4 -rw-r--r--   1 josip    josip         403 Mar 17 11:27 salt/ext_pillar/hosts/minion-name/userdata/stuff/id_rsa.pub
 14550    4 drwxr-xr-x   2 josip    josip        4096 Mar 22 11:52 salt/ext_pillar/hosts/minion-name/ssl-certificates
 14558    4 -rw-r-----   1 josip    josip        1708 Sep 18  2015 salt/ext_pillar/hosts/minion-name/ssl-certificates/star.dom.ain.2015-09-21.key
 14557    4 -rw-r--r--   1 josip    josip        1822 Sep 22  2015 salt/ext_pillar/hosts/minion-name/ssl-certificates/star.dom.ain.2015-09-21.crt
 14555    4 -rw-r--r--   1 josip    josip        1647 Sep 21  2015 salt/ext_pillar/hosts/minion-name/ssl-certificates/star.dom.ain.chain.crt
 14556    4 -rw-r--r--   1 josip    josip        3469 Sep 21  2015 salt/ext_pillar/hosts/minion-name/ssl-certificates/star.dom.ain.2015-09-21.bundle.crt
 14577    4 -rw-r--r--   1 josip    josip         377 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/more-stuff.conf
 14561    4 drwxr-xr-x   2 josip    josip        4096 Apr  5 19:06 salt/ext_pillar/hosts/minion-name/nginx
 14566    8 -rw-r--r--   1 josip    josip        4303 Mar 22 15:31 salt/ext_pillar/hosts/minion-name/nginx/local-stuff.conf

Steps to Reproduce Issue

% time salt-ssh -c /opt/stuff/salt minion-name saltutil.refresh_pillar
minion-name:
    False
salt-ssh -c /opt/stuff/salt    0.28s user 0.11s system 8% cpu 4.969 total

Versions Report

% salt-ssh --versions-report
Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-108-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[Ch3LL]

I am able to replicate this. One of the great ways to troubleshoot salt-ssh is to run it with -ltrace. It will show you the exact commands it runs on the minion. I believe this is expected behavior that you are getting a False because salt-ssh runs with salt-call --local on the minion and can't publish back to a master to refresh pillar data.

But I would like to get @terminalmage 's input here to make sure I am correct on this. I could be wrong and we need to fix this saltutil call for salt-ssh.

[terminalmage]

My best guess is that this is happening because saltutil.refresh_pillar fires an event (using event.fire) on the minion bus so that the minion can asynchronously handle the refresh. The return value for saltutil.refresh_pillar is the return from the event.fire, which is True/False based whether or not the minion was successfully able to fire the event on the minion bus. Since salt-ssh has no minion daemon, there is no minion bus, and the event cannot successfully be fired. Hence, a False return.

It's unclear to me why you're running saltutil.refresh_pillar for salt-ssh in the first place though. Are you not seeing changes to pillar reflected in your salt-ssh calls? As I understand it, salt-ssh compiles pillar for each run, so there should never be a need to refresh it.

[shallot]

@terminalmage all the documentation I encountered that referred to pillars seemed fairly adamant in that you have to run the refresh, no buts. I searched again to verify and saw such references to that in:

https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html

"To ensure that the minions have the new pillar data, issue a command to them asking that they fetch their pillars from the master"

https://docs.saltstack.com/en/latest/topics/pillar/

"This function triggers the minion to asynchronously refresh the in-memory pillar data and will always return None."

"So, if pillar data is modified, and then states are run, the states will see the updated pillar data, but pillar.item, pillar.get, and pillar.raw will not see this data unless refreshed using saltutil.refresh_pillar."

https://docs.saltstack.com/en/latest/topics/development/external_pillars.html

"Just as with traditional pillars, external pillars must be refreshed in order for minions to see any fresh data"

[shallot]

@Ch3LL it would be nice if we didn't have to go all the way to trace level to get that info with salt-ssh. Maybe if only those messages were bumped up to level debug in case salt-ssh was being run, those that mention SALT_ARGV? It looks like salt/client/ssh/ssh_py_shim.py shows this on its standard error. Whatever runs that seems to stash that stderr to trace level?

[terminalmage]

@shallot All of that applies to minions who run using a master, since pillar is compiled on the master for security. Masterless minions refresh pillar on every run. This is probably something we should document better.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[shallot]

The docs appear unchanged, this stands

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[barbaricyawps]

All of that applies to minions who run using a master, since pillar is compiled on the master for security. Masterless minions refresh pillar on every run. This is probably something we should document better.

I am unclear what the specific documentation task is here. Is the ask to update the pillar documentation to mention that masterless minions refresh pill on every run on one of these topics?

https://docs.saltstack.com/en/latest/topics/tutorials/pillar.html https://docs.saltstack.com/en/latest/topics/pillar/ https://docs.saltstack.com/en/latest/topics/development/external_pillars.html

For anyone on this thread, please feel free to clarify the specific action needed to bring this docs issue to a close. Without further clarification, I might need to close this ticket as being unactionable.

[whytewolf]

closing out enough time has passed and no comments have been made.