saltenv not used from roster with salt-ssh state.apply

vbasem commented 6 years ago

Description of Issue/Question

I am trying to define saltenv in my roster files and have the correct environment applied when salt-ssh is called. What I observe is state.apply always seems to default to "base" environment, which pillar.items uses the correct saltenv defined in the roster.

Setup

roster

  user: root
  host: localhost
  minion_opts:
    environment: cloud_local
    saltenv: cloud_local
    pillarenv_from_saltenv: True

/etc/salt/master

pillarenv_from_saltenv: True
pillar_source_merging_strategy: none

file_roots:
  cloud_local:
    - /srv/salt/cloud/
pillar_roots:
    - /srv/pillar/cloud/local

test.sls

doit:
  cmd.run:
    - name: echo {{ saltenv }}

top.sls

{{saltenv}}
  '*':
    - test

Steps to Reproduce Issue

1) salt ssh will output base (expected is cloud_local)

salt-ssh localhost state.apply test

2) salt ssh will output cloud_local

salt-ssh localhost state.apply test saltenv=cloud_local

1) salt call will output cloud_local (put saltenv: cloud_local in minion file)

salt-call state.apply test

Versions Report

(master, minion and ssh are all the same version)

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.26.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

gtmanfred commented 6 years ago

Yup, this is a bug with saltenv, the state.sls wrappers default to base and do not look at the minion_opts for knowing if a saltenv is specified.

Thanks for reporting. Daniel

Arendtsen commented 6 years ago

Would this bug also affect salt-call? I have the exact same situation.

mbochenk commented 5 years ago

Same issue with salt-ssh.

if I set up saltenv in roster, it does not work
if pillarenv_from_saltenv: True seems to apply to pillarenv (saltenv remains base)
if I add saltenv=some_env to invocation - works

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: 1.3.8
         Jinja2: 2.7.2
        libgit2: 0.26.3
        libnacl: 1.6.1
       M2Crypto: 0.28.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: 0.4.3
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 4.4.163-1.el7.elrepo.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

gtmanfred commented 5 years ago

@saltstack/team-triage can you take a look at this?

rajkrish commented 1 year ago

This issue is still affecting me in the latest salt-ssh version (that is, saltenv specified in minion_opts in roster.yaml not being recognized) -

# salt-ssh --version
salt-ssh 3005.1

Rastagong commented 5 months ago

Can also confirm on salt-ssh 3007.1 (Chlorine).

In my case, I was not using custom file_roots in the minion_opts of the roster.
I wasn't trying to have the correct saltenv applied to the entire SLS state.

Instead, I was trying to pass a salt:// URL with a specific saltenv using the saltenv query string syntax (?saltenv=), like this:

A test file sourced from a custom saltenv:
  file.managed:
    - name: /tmp/test_file
    - source: salt://default_server/default.key?saltenv=my_custom_saltenv

But it didn't work, for the same reason highlighted above (the salt-ssh state.sls wrapper defaults only to the "base" saltenv, incriminating line here I think?).
So I kept getting Source file salt://default_server/default.key?saltenv=my_custom_saltenv not found in saltenv 'base' errors.

A quick and dirty fix if, like me, you're only trying to use a salt:// URL with a ?saltenv= query string syntax to fetch files from a specific saltenv using salt-ssh. Replace the lowstate_file_refs function in salt/client/ssh/state.py file (/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/ssh/state.py for me) with the following:

def lowstate_file_refs(chunks, extras=""):
    """
    Create a list of file ref objects to reconcile
    """
    refs = {}
    for chunk in chunks:
        if not isinstance(chunk, dict):
            continue
        saltenv = "base"
        crefs = []
        for state in chunk:
            if state == "__env__":
                saltenv = chunk[state]
            elif state.startswith("__"):
                continue
            crefs.extend(salt_refs(chunk[state]))
        if saltenv not in refs:
            refs[saltenv] = []
        if crefs:
            refs[saltenv].append(crefs)
    #log.error(f"{crefs=}")
    #log.error(f"{crefs=}")
    if extras:
        extra_refs = extras.split(",")
        if extra_refs:
            for env in refs:
                for x in extra_refs:
                    refs[env].append([x])
    try: #START of the quick and dirty fix
        for env in tuple(refs):
            for i,sublist in enumerate(tuple(refs[env])):
                for ii,url_string in enumerate(sublist):
                    log.error(f"For env {env} sublist no {i} item no {ii}, there is URL {url_string}")
                    path, env_new = salt.utils.url.parse(url_string)
                    if env_new is not None:
                        path_final = f"salt://{path}"
                        log.error(f"URL {url_string} should be read as URL {path_final} of saltenv {env_new}")
                        refs[env][i].remove(url_string)
                        if env_new not in refs:
                            refs[env_new] = []
                        refs[env_new].append([path_final])
            refs[env] = [sublist for sublist in refs[env] if sublist]
        refs = {env:refs[env] for env in refs if any(refs[env])}
    except Exception as error:
        log.exception(error)
        raise error from error
    #END of the quick and dirty fix
    return refs

The only part which I added is the try/except block at the end, just above the final return refs line. I included a few error-level logs so you can check which salt:// URLs are being transformed.

saltstack / salt