Salt-SSH 2018.3 Permission denied: '/var/tmp/.xyz/running_data/var'

MatthiasKuehneEllerhold commented 6 years ago

Description of Issue/Question

After the upgrade to 2018.3.0 our Salt-SSH setup throws this error:

$ salt-ssh my-server.org apply test=True
[ERROR   ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.xyz/running_data/var'
my-server.org:
    An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.xyz/running_data/var'

This happens on every minion. The directory /var/tmp/.xyz/running_data is only readable by root, not by the salt-user. But this seems to be correct (comparing to older directories).

Downgrading to 2017.7.4+ds-1 works. Steps:

Add "deb http://repo.saltstack.com/apt/debian/9/amd64/archive/2017.7.4/ stretch main" to your apt sources list.
apt-get update
apt-get install salt-ssh=2017.7.4+ds-1 salt-common=2017.7.4+ds-1

Setup

Standard Salt-SSH setup with a master and some minions.

Steps to Reproduce Issue

$ salt-ssh my-server.org state.apply test=True

Versions Report

$ salt-ssh --versions-report
Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Nov 24 2017, 17:33:09)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: Not Installed
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: Not Installed

System Versions:
           dist: debian 9.4
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-6-amd64
         system: Linux
        version: debian 9.4

sahama commented 6 years ago

i have this problem too. i install salt from cloned git repo by pip install -e on python 3.4 Salt Version: Salt: 2018.3.0-339-g5852993

Dependency Versions: cffi: 1.11.2 cherrypy: Not Installed dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: 1.0.7 msgpack-pure: Not Installed msgpack-python: 0.4.8 mysql-python: Not Installed pycparser: 2.18 pycrypto: 2.6.1 pycryptodome: 3.4.7 pygit2: Not Installed Python: 3.4.2 (default, Oct 8 2014, 10:45:20) python-gnupg: 0.4.2 PyYAML: 3.12 PyZMQ: 16.0.3 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.5.2 ZMQ: 4.0.5

System Versions: dist: debian 8.10 locale: UTF-8 machine: x86_64 release: 3.16.0-5-amd64 system: Linux version: debian 8.10

jos666 commented 6 years ago

look the file: /etc/salt/roster User not root, or not in /etc/sudoers, or not use sudo: True

e.g /etc/salt/roster :

root user

web1:
host: 192.168.0.21
user: root
passwd: test

not root other user, user in /etc/sudoers (abc ALL=(ALL) NOPASSWD: ALL)
```
web1:
host: 192.168.0.5
user: abc
passwd: test
sudo: True
```

MatthiasKuehneEllerhold commented 6 years ago

@jos666 It works in version 2017.7.4 where all of this was already necessary.

We're not using the root user.
The user is in the sudoers file correctly.
"sudo: True" is in the roster.

danbst commented 6 years ago

Same problem after upgrade to 18.03

terminalmage commented 6 years ago

I can't reproduce this. Can one of you provide an example SLS file that triggers this error?

danbst commented 6 years ago

Indeed, it is hard to reproduce. Here is listing of a directory when I run fresh 18.3.0:

$ ls -la /var/tmp/._ubuntu_dd8a91_salt/
total 64
drwxrwx--- 4 root   ubuntu  4096 Apr 18 11:56 .
drwxrwxrwt 5 root   root    4096 Apr 18 23:48 ..
-rw-rw-r-- 1 ubuntu ubuntu    40 Oct 18 14:10 ext_version
-rw-r--r-- 1 root   root     253 Apr 18 12:54 minion
drwx------ 9 root   root    4096 Apr 18 11:56 py2
drwx------ 3 root   root    4096 Apr 18 11:56 running_data
-rw-rw-r-- 1 ubuntu ubuntu   231 Apr 18 11:36 salt-call
-rw------- 1 ubuntu ubuntu 27204 Apr 18 12:54 salt_state.tgz
-rw-rw-r-- 1 ubuntu ubuntu     1 Apr 18 11:36 .thin-gen-py-version
-rw-rw-r-- 1 ubuntu ubuntu     8 Apr 18 11:36 version

When I remove this directory and rerun state.highstate, the listing is like this:

$ ls -la /var/tmp/.ubuntu_dd8a91_salt/
total 12
drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 18 23:48 .
drwxrwxrwt 5 root   root   4096 Apr 18 23:48 ..
drwxrwxr-x 3 ubuntu ubuntu 4096 Apr 18 23:48 running_data

Deployment works now (notice owner of running_data has changed). So I think this is cache issue - new Salt doesn't clear previous Salt (17.7.4) cache directory before deployment.

MatthiasKuehneEllerhold commented 6 years ago

I've tested this extensivly. Neither clearing the cache on the minion (/var/tmp/) nor on the master (as defined by the key "cachedir" in my master-config) changes anything.

Here is the directory listing of a (working!) 2017.7.4:

$ ls -lA /var/tmp/.salt_a40eac_salt
-rw-r--r-- 1 root root  196 Apr 16 14:03 minion
drwx------ 9 root root 4,0K Apr  6 07:20 py2
drwxr-x--- 3 root root 4,0K Apr  6 07:20 running_data
-rw-r--r-- 1 salt salt  231 Apr  5 14:08 salt-call
-rw-r--r-- 1 salt salt    1 Apr  5 14:08 .thin-gen-py-version
-rw-r--r-- 1 salt salt    8 Apr  5 14:08 version

My salt user is named "salt" btw.

MatthiasKuehneEllerhold commented 6 years ago

I've boiled the error down to this minimally reproducible state.

testing/test.sls:

test-write-file:
  file.managed:
    - name:      /root/my-test-file.txt
    - source:    salt://testing/files/my-test-file.txt

testing/files/my-test-file.txt:

This should work.

roster:

localhost:
  host: 192.168.x.y
  user: salt
  sudo: True
  minion_opts:
    backup_mode: minion

Calling salt-ssh

$ rm -Rf cache
$ rm -Rf /var/tmp/.salt_a40eac_salt
$ salt-ssh localhost state.apply testing.test
[ERROR   ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.salt_a40eac_salt/running_data/var'
localhost:
    An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.salt_a40eac_salt/running_data/var'

So it seems like a file must be involved for it to fail.

Just calling salt-ssh localhost cmd.run "date" works. The same goes for simple states that use cmd.run.

Downgrading to 2017.7.4 and running the exact same thing works.

danbst commented 6 years ago

@MatthiasKuehneEllerhold How do you invoke salt-ssh 18.3.0? Do you install it globally in your distro, or install it in separate virtualenv? I've noticed the issue exists when salt is installed globally, but no issue when run from virtualenv. My distro is Ubuntu 16.04 though.

MatthiasKuehneEllerhold commented 6 years ago

I've installed it via apt-get (on Debian Stretch) globally.

danbst commented 6 years ago

Can you try to run it from virtualenv? If it works there, then it's distribution issue.

$ virtualenv venv-salt
$ source venv-salt/bin/activate
$ pip install salt
$ salt-ssh -c ....

terminalmage commented 6 years ago

I was running via a pip-installed salt-ssh, I used a fresh Debian container and installed 2017.7.4 from repo.saltstack.com, and ran @MatthiasKuehneEllerhold's testing.test SLS file, then upgraded to 2018.3.0 from the upstream debian repo and was unable to reproduce the error.

@MatthiasKuehneEllerhold If you're using sudo: True then salt-ssh will sudo to root and there should be no possible way that this error shows up.

MatthiasKuehneEllerhold commented 6 years ago

I've added my roster file to my previous comment. Yes I'm using sudo: True and it works perfectly on 2017.7.4.

I've ran the same command with debug-level logging. Here is the last few lines directly before the error:

$ salt-ssh -ldebug localhost state.apply testing.test
[DEBUG   ] Rendered data from file: /home/salt/saltstack/cache/files/base/testing/test.sls:
test-write-file:
  file.managed:
    - name:      /root/my-test-file.txt
    - source:    salt://testing/files/my-test-file.txt

[DEBUG   ] Results of YAML rendering: 
OrderedDict([(u'test-write-file', OrderedDict([(u'file.managed', [OrderedDict([(u'name', u'/root/my-test-file.txt')]), OrderedDict([(u'source', u'salt://testing/files/my-test-file.txt')])])]))])
[PROFILE ] Time (in seconds) to render '/home/salt/saltstack/cache/files/base/testing/test.sls' using 'yaml' renderer: 0.00156784057617
[ERROR   ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.salt_a40eac_salt/running_data/var'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/__init__.py", line 1137, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 462, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 174, in sls
    roster_grains)
  File "/usr/lib/python2.7/dist-packages/salt/client/ssh/state.py", line 218, in prep_trans_tar
    path = file_client.cache_file(name, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 195, in cache_file
    path, '', True, saltenv, cachedir=cachedir, source_hash=source_hash)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 517, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1091, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1346, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1339, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1331, in __hash_and_stat_file
    return self.channel.send(load)
  File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 870, in send
    return getattr(self.fs, cmd)(load)
  File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 644, in file_hash
    return self.__file_hash_and_stat(load)[0]
  File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 636, in __file_hash_and_stat
    return self.servers[fstr](load, fnd), stat_result
  File "/usr/lib/python2.7/dist-packages/salt/fileserver/roots.py", line 275, in file_hash
    os.makedirs(cache_dir)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/var/tmp/.salt_a40eac_salt/running_data/var'
[DEBUG   ] LazyLoaded nested.output
localhost:
    An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.salt_a40eac_salt/running_data/var'

sahama commented 6 years ago

i found this problem will happen when i use sudo user instead root and i am trying to copy a file to minion (for example via file.managed).

gtmanfred commented 6 years ago

I am unable to replicate this as well. Can you take a look at this, and tell me what I can do to replicate the issue?

The only way I was able to replicate the issue was by starting with sudo: True in my roster, and syncing everything over, so it gets owned by the root user, and then removeing the sudo: true, which would try to write the files as the regular user.

https://gist.github.com/gtmanfred/6166f7d74a05c589a0fd06335e64f1dc

terminalmage commented 6 years ago

If sudo: True is set, we login as the unprivilieged user and then sudo to root. Maybe your /etc/sudoers is not properly configured?

sahama commented 6 years ago

i found a way to reproduce it. and during it i found another bug :D i create and active a virtual env of python3.4 in /home/sahama/projects/env i clone development branch in to /home/sahama/projects/salt i configured _syspath like this:

ROOT_DIR = '/home/sahama/projects/salt_root'
SHARE_DIR = None
CONFIG_DIR = None
CACHE_DIR = None
SOCK_DIR = None
SRV_ROOT_DIR = None
BASE_FILE_ROOTS_DIR = None
BASE_PILLAR_ROOTS_DIR = None
BASE_MASTER_ROOTS_DIR = None
BASE_THORIUM_ROOTS_DIR = None
LOGS_DIR = None
PIDFILE_DIR = None
SPM_FORMULA_PATH = None
SPM_PILLAR_PATH = None
SPM_REACTOR_PATH = None
HOME_DIR = None

i create a roster file in /home/sahama/projects/salt_root/etc/salt/roster link this:

localhost:
    host: localhost

i create a statefile in ~/projects/salt_root/srv/salt/test.sls

create test file:
  file.managed:
    - name: /root/delme
    - source: salt://sample_file
    - template: jinja

and top file in ~/projects/salt_root/srv/salt/top.sls:

base:
  '*':
    - test

and sample file in ~/projects/salt_root/srv/salt/sample_file contains sample content.

i configured master in ~/projects/salt_root/etc/salt/master.d/main.conf like this:

publisher_acl:
  sahama:
    - .*

file_recv: True

i remove all salt env with:

sudo rm -rf /var/tmp/.*salt

now i use this command for test:

salt-ssh localhost --user=sahama --passwd=1 --sudo cmd.run whoami

and result is :

localhost:
    root

and finally i use this command:

salt-ssh localhost --user=sahama --passwd=1 --sudo state.apply test

and this is what i see:

[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
[CRITICAL] Unable to make cachedir /var/tmp/.sahama_659579_salt/running_data/var/cache/salt/minion/file_lists/roots
localhost:
----------
          ID: create test file
    Function: file.managed
        Name: /root/delme
      Result: False
     Comment: Source file salt://sample_file not found in saltenv 'base'
     Started: 07:23:12.706619
    Duration: 71.848 ms
     Changes:   

Summary for localhost
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:  71.848 ms

gtmanfred commented 6 years ago

You should not be modifying root_dir in the _syspaths.py file.

To configure root_dir, you should specify it in the minion file inside of the --config-dir that you are specifying on the command line.

Are you still able to replicate this without manually setting defaults in the code?

MatthiasKuehneEllerhold commented 6 years ago

My sudoers looks like this:

$ cat /etc/sudoers.d/salt-sudo
# User salt may run any command with sudo without needing a password
salt ALL=(ALL) NOPASSWD: ALL

I've tested it multiple times:

sudo -i works
salt-ssh 2017.7.4 works

I didnt change anything in _syspath (I'm not even sure where this is...), but I'm guessing its similar to this file on my salt-master:

$ cat /home/salt/saltstack-root/config/master.d/paths.conf
file_roots:
  base:
    - /home/salt/saltstack-root/states/
pillar_roots:
  base:
    - /home/salt/saltstack-root/pillar/
root_dir: /home/salt/saltstack-root/
ssh_log_file: /home/salt/saltstack-root/logs/salt-ssh.log
pki_dir: /home/salt/saltstack-root/pki/
cachedir: /home/salt/saltstack-root/cache/

theWizK commented 6 years ago

I'm having the same issue after upgrading to salt-ssh-2018.3.2-1.el7 (rpm version). I also tried executing from virtualenv:

Salt Version: Salt: 2018.3.2

Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.10 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.5 (default, Apr 11 2018, 07:36:10) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: Not Installed RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 5.1 ZMQ: Not Installed

System Versions: dist: centos 7.4.1708 Core locale: UTF-8 machine: x86_64 release: 3.10.0-693.5.2.el7.x86_64 system: Linux version: CentOS Linux 7.4.1708 Core

This also worked for me on earlier version(s) of salt-ssh.

My Saltfile (in my local directory) looks like this:

salt-ssh: log-level: quiet roster_file: /etc/salt/roster config_dir: ../salt-ssh/config ssh_log_file: ../salt-ssh/log/salt-ssh.log ssh_priv: agent-forwarding ssh_max_procs: 10 ssh_wipe: True ssh_rand_thin_dir: True ssh_user: centos ssh_sudo: True ssh_tty: True

My master config from ../salt-ssh/config looks like this:

cachedir: /srv/salt-ssh/tmp

file_roots: base:

/srv/salt

pillar_roots: base:

/srv/pillar-ssh

roster_defaults: user: centos sudo: true

my roster file only contains a few grains for the host.

execution looks like this:

(salttest) [theWizK@auto1 salt]$ salt-ssh vhost-01 state.apply ministack/provision [ERROR ] An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.centos_8fdaca_salt/running_data/var/cache/salt/minion/roots/hash/base/ministack' vhost-01: An Exception occurred while executing state.apply: [Errno 13] Permission denied: '/var/tmp/.centos_8fdaca_salt/running_data/var/cache/salt/minion/roots/hash/base/ministack'

I can ssh in to my host as centos@vhost-01 and successfully sudo -i with no password.

lloesche commented 6 years ago

I'm seeing the same with salt-ssh 2018.3.2 (Oxygen).

My roster

bootstrap:
  host: localhost
  user: lukas
  sudo: True

Saltfile

salt-ssh:
  config_dir: ./etc/salt
  max_procs: 30
  wipe_ssh: False

The directory salt-ssh creates has messed up permissions. Some of the files are owned by root and some by the unprivileged user lukas which then seems to try and access some of the data owned by root:

lukas@bigmac:~/salt-ssh-test$ ls -al /var/tmp/.lukas_6f44fe_salt/
total 0
drwxrwx--- 1 root  lukas 4096 Oct 13 14:28 .
drwxrwxrwt 1 root  root  4096 Oct 13 14:27 ..
-rw-r--r-- 1 lukas lukas   40 Oct 13 14:16 ext_version
-rw-r--r-- 1 root  root   161 Oct 13 14:28 minion
drwx------ 1 root  root  4096 Oct 13 14:28 py2
drwx------ 1 root  root  4096 Oct 13 14:28 running_data
-rw-r--r-- 1 lukas lukas  231 Oct 13 14:16 salt-call
-rw-r--r-- 1 lukas lukas    1 Oct 13 14:16 .thin-gen-py-version
-rw-r--r-- 1 lukas lukas    8 Oct 13 14:16 version

This only happens in a state using file.managed. Other states work just fine.

The complete state

{% from 'dcos/_clusters.sls' import clusters, cache_dir %}

{% for cluster, properties in clusters.items() %}
{{ cluster }} installer download:
  file.managed:
    - name: {{ cache_dir }}/{{ cluster }}/dcos_generate_config.sh
    - source: {{ properties['installer']['url'] }}
    - source_hash: {{ properties['installer']['hash'] }}
    - makedirs: True
    - mode: 755
{% endfor %}

The complete error

bootstrap:
    - Rendering SLS 'base:dcos.genconf' failed: Jinja error: [Errno 13] Permission denied: '/var/tmp/.lukas_6f44fe_salt/running_data/var'
      Traceback (most recent call last):
        File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 389, in render_jinja_tmpl
          output = template.render(**decoded_context)
        File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 1008, in render
          return self.environment.handle_exception(exc_info, True)
        File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 780, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "<template>", line 1, in top-level template code
        File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 109, in get_source
          self.check_cache(template)
        File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 97, in check_cache
          self.cache_file(template)
        File "/usr/lib/python2.7/dist-packages/salt/utils/jinja.py", line 90, in cache_file
          self.file_client().get_file(saltpath, '', True, self.saltenv)
        File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1089, in get_file
          hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
        File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1344, in hash_and_stat_file
          hash_result = self.hash_file(path, saltenv)
        File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1337, in hash_file
          return self.__hash_and_stat_file(path, saltenv)
        File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1329, in __hash_and_stat_file
          return self.channel.send(load)
        File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 871, in send
          return getattr(self.fs, cmd)(load)
        File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 644, in file_hash
          return self.__file_hash_and_stat(load)[0]
        File "/usr/lib/python2.7/dist-packages/salt/fileserver/__init__.py", line 636, in __file_hash_and_stat
          return self.servers[fstr](load, fnd), stat_result
        File "/usr/lib/python2.7/dist-packages/salt/fileserver/roots.py", line 275, in file_hash
          os.makedirs(cache_dir)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 150, in makedirs
          makedirs(head, mode)
        File "/usr/lib/python2.7/os.py", line 157, in makedirs
          mkdir(name, mode)
      OSError: [Errno 13] Permission denied: '/var/tmp/.lukas_6f44fe_salt/running_data/var'

      ; line 1

      ---
      {% from 'dcos/_clusters.sls' import clusters, cache_dir %}    <======================

      include:
        - .docker
        - .config
        - .download
      [...]
      ---

MatthiasKuehneEllerhold commented 6 years ago

With the repository pinned to version 2017.7.4 I didnt get any new version of salt-ssh. Today I've tested some:

2017.7.4 works
2017.7.5 error
2017.7.6 error
2017.7.7 error
2017.7.8 error
2018.3.1 error
2018.3.2 error
2018.3.3 error

Does this help in determining what caused this error?

Using 2017.7.5 as a testbed: Executing chmod -R 0777 /var/tmp/.salt_a40eac_salt/running_data on the minion fixes the problem. Executing chown -R salt /var/tmp/.salt_a40eac_salt/running_data on the minion fixes the problem too.

("salt" is my salt-ssh user...)

deniswal commented 5 years ago

salt-ssh leaves a bunch of root-owned files in /vat/tmp/ (user vagrant for me), at the next execution ->

[ERROR ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.vagrant_60a1e1_salt/running_data/var'.

I am hurt by this bug too.

CentOS7 salt-ssh 2018.3.3 (Oxygen) Running salt as user vagrant, sudo: True (sudoer :NOPASSWD)

Ch3LL commented 5 years ago

i was finally able to replicate this after many attempts using @MatthiasKuehneEllerhold 's test case.

I was able to bisect it to this commit:

a8cded1c836198c1ad7a6f521419b9ef10ae06d7

which is from this PR https://github.com/saltstack/salt/pull/46149

ping @terminalmage any quick ideas here by chance?

terminalmage commented 5 years ago

No, unfortunately. Can you get this case into a docker container that I could use to reproduce?

dbauman commented 5 years ago

Edit: I put the source for a docker image here: https://github.com/dbauman/salt-ssh-46891.git

I've seen this problem and believe it's the result of code on running on the master using the cachedir specified for the minion.

To make the problem easier to see, you might want to specify a thin_dir that isn't writable by the user running salt-ssh:

[saltadm@myhost ~]$ cat etc/salt/roster 
dummy:
   host: localhost
   user: minion
   sudo: True
   tty: True
   thin_dir: /home/minion/salt-deploy
[saltadm@myhost ~]$ salt-ssh --version
salt-ssh 2019.2.0 (Fluorine)
[saltadm@myhost ~]$ python --version
Python 2.7.14
[saltadm@myhost ~]$ cat srv/salt/localtest.sls 
run_test:
   cmd.run:
      - name: id -un
[saltadm@dpydalctadm01 ~]$ salt-ssh -i 'dummy' state.apply localtest
[CRITICAL] Unable to make cachedir /home/minion/salt-deploy/running_data/var/cache/salt/minion/file_lists/roots
...

I can make the problem go away with the following change:

Note -- I have neither the knowledge nor the test cases to speculate as to whether this horribly breaks something else or if other fields should be copied from __master_opts__

[saltadm@dpydalctadm01 ~]$ diff salt-save/salt/client/ssh/state.py .pyenv/versions/2.7.14/lib/python2.7/site-packages/salt/client/ssh/state.py
75,76c75,89
<         salt.state.BaseHighState.__init__(self, opts)
<         self.state = SSHState(opts, pillar, wrapper)
---
>         lopts={}
>         lopts.update(opts)
>         if '__master_opts__' in opts:
>             mo=opts['__master_opts__']
>             if 'cachedir' in mo:
>                 log.debug('XXX o.cachedir=%s m.cachedir=%s' %
>                           (opts['cachedir'], mo['cachedir']) )
>                 lopts['cachedir']=mo['cachedir']
>             else:
>                 log.debug('XXX no cachdir in master_opts')
>         else:
>             log.debug('XXX No master opts')
> 
>         salt.state.BaseHighState.__init__(self, lopts)
>         self.state = SSHState(lopts, pillar, wrapper)

non7top commented 5 years ago

I'm seeing the same issue

[DEBUG   ] Could not find file 'salt://_modules' in saltenv 'base'
[INFO    ] Caching directory '_modules/' for environment 'base'
[CRITICAL] Unable to make cachedir /var/tmp/.deploy_d1fd54_salt/running_data/var/cache/salt/minion/file_lists/roots

(local invoking user = non7top, remote ssh&sudo user = deploy, using the same host) Permissions on that dir are an assortment of all users.

$ sudo ls -l /var/tmp/.deploy_d1fd54_salt/ -la
total 48
drwxrwx--- 5 root    deploy  4096 Jun 30 12:32 .
drwxrwxrwt 6 root    root    4096 Jun 30 12:32 ..
-rw-rw-r-- 1 non7top non7top   65 Jun 30 12:32 code-checksum
-rw-r--r-- 1 root    root     153 Jun 30 12:32 minion
drwx------ 5 root    root    4096 Jun 30 12:32 py3
drwx------ 6 root    root    4096 Jun 30 12:32 pyall
drwx------ 3 root    root    4096 Jun 30 12:32 running_data
-rw-rw-r-- 1 non7top non7top  757 Jun 30 12:32 salt-call
-rw------- 1 deploy  deploy  1884 Jun 30 12:32 salt_state.tgz
-rw-rw-r-- 1 non7top non7top    8 Jun 30 12:32 supported-versions
-rw-rw-r-- 1 non7top non7top    1 Jun 30 12:32 .thin-gen-py-version
-rw-rw-r-- 1 non7top non7top    8 Jun 30 12:32 version

Not quite sure which process(and uid) generates that. Either a local user that is running salt-ssh. but not sure why it would ever do it. Or remote user without sudo, which now runs something without sudo. Apparently the latter.

deniswal commented 5 years ago

Still not solved with: salt-ssh 2019.2.0 (Fluorine)

deniswal commented 5 years ago

[ERROR ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.ubuntu_af974b_salt/running_data/var' Occurs at the second run (The first one is OK) The state applied involves pkg installation using sudo: True on the minion side.

deniswal commented 5 years ago

[ERROR ] An Exception occurred while executing state.sls: [Errno 13] Permission denied: '/var/tmp/.ubuntu_af974b_salt/running_data/var' Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/salt/client/ssh/init.py", line 1159, in run_wfunc result = self.wfuncs[self.fun](*self.args, **self.kwargs) File "/usr/lib/python2.7/dist-packages/salt/client/ssh/wrapper/state.py", line 186, in sls highdata, errors = st.render_highstate({saltenv: mods}) File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3804, in render_highstate sls, saltenv, mods, matches) File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3604, in render_state matches File "/usr/lib/python2.7/dist-packages/salt/state.py", line 3456, in render_state state_data = self.client.get_state(sls, saltenv) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 371, in get_state dest = self.cache_file(path, saltenv, cachedir=cachedir) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 194, in cache_file path, '', True, saltenv, cachedir=cachedir, source_hash=source_hash) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 475, in get_url result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1053, in get_file hash_server, stat_server = self.hash_and_stat_file(path, saltenv) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1312, in hash_and_stat_file hash_result = self.hash_file(path, saltenv) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1305, in hash_file return self.hash_and_stat_file(path, saltenv) File "/usr/lib/python2.7/dist-packages/salt/fileclient.py", line 1297, in __hash_and_stat_file return self.channel.send(load) File "/usr/lib/python2.7/dist-packages/salt/fileserver/init.py", line 906, in send return getattr(self.fs, cmd)(load) File "/usr/lib/python2.7/dist-packages/salt/fileserver/init.py", line 681, in file_hash return self.__file_hash_and_stat(load)[0] File "/usr/lib/python2.7/dist-packages/salt/fileserver/init.py", line 673, in file_hash_and_stat return self.servers[fstr](load, fnd), stat_result File "/usr/lib/python2.7/dist-packages/salt/fileserver/roots.py", line 273, in file_hash os.makedirs(cache_dir) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 150, in makedirs makedirs(head, mode) File "/usr/lib/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 13] Permission denied: '/var/tmp/.ubuntu_af974b_salt/running_data/var'

Ch3LL commented 5 years ago

ive added this issue to the workgroup salt-ssh's backlog, but in the meantime anyone is welcome to dive in and submit a PR.

nmalbran commented 5 years ago

If you're still struggling with this, you can run salt-ssh with -W option: -W, --rand-thin-dir Select a random temp dir to deploy on the remote system. The dir will be cleaned after the execution.

einsiedlerkrebs commented 4 years ago

I have the same issue.

If you're still struggling with this, you can run salt-ssh with -W option: -W, --rand-thin-dir Select a random temp dir to deploy on the remote system. The dir will be cleaned after the execution.

But -W option does not help. It started when I wanted to use thin_dir in roster the first time. Upgrade to salt3000 did not help the issue.

    - Rendering SLS init.set_grains_py failed, render error: [Errno 13] Permission denied: '/var/tmp/.20b004'
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/salt/state.py", line 3601, in render_state
          rendered_sls=mods
        File "/usr/lib/python3/dist-packages/salt/template.py", line 101, in compile_template
          ret = render(input_data, saltenv, sls, **render_kwargs)
        File "/usr/lib/python3/dist-packages/salt/renderers/pyobjects.py", line 501, in render
          final_template, final_globals = process_template(template)
        File "/usr/lib/python3/dist-packages/salt/renderers/pyobjects.py", line 458, in process_template
          state_file = client.cache_file(import_file, saltenv)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 193, in cache_file
          path, '', True, saltenv, cachedir=cachedir, source_hash=source_hash)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 474, in get_url
          result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1066, in get_file
          hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1325, in hash_and_stat_file
          hash_result = self.hash_file(path, saltenv)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1318, in hash_file
          return self.__hash_and_stat_file(path, saltenv)
        File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1310, in __hash_and_stat_file
          return self.channel.send(load)
        File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 906, in send
          return getattr(self.fs, cmd)(load)
        File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 681, in file_hash
          return self.__file_hash_and_stat(load)[0]
        File "/usr/lib/python3/dist-packages/salt/fileserver/__init__.py", line 673, in __file_hash_and_stat
          return self.servers[fstr](load, fnd), stat_result
        File "/usr/lib/python3/dist-packages/salt/fileserver/roots.py", line 281, in file_hash
          os.makedirs(cache_dir)
        File "/usr/lib/python3.7/os.py", line 211, in makedirs
          makedirs(head, exist_ok=exist_ok)
        File "/usr/lib/python3.7/os.py", line 211, in makedirs
          makedirs(head, exist_ok=exist_ok)
        File "/usr/lib/python3.7/os.py", line 211, in makedirs
          makedirs(head, exist_ok=exist_ok)
        [Previous line repeated 7 more times]
        File "/usr/lib/python3.7/os.py", line 221, in makedirs
          mkdir(name, mode)
      PermissionError: [Errno 13] Permission denied: '/var/tmp/.20b004'

baby-gnu commented 4 years ago

I have the same issue with my salt-ssh configuration.

I must remove the thin_dir from my roster (it's working with /var/tmp/) otherwise fails with the following error:

[CRITICAL] Rendering SLS 'base:salt.pkgrepo.debian.install' failed: Jinja error: [Errno 13] Permission denied: '/root/.cache'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 392, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 3, in <module>
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 158, in get_source
    self.check_cache(_template)
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 119, in check_cache
    self.cache_file(template)
  File "/usr/lib/python3/dist-packages/salt/utils/jinja.py", line 112, in cache_file
    self.file_client().get_file(saltpath, '', True, self.saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1103, in get_file
    with self._cache_loc(
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 154, in _cache_loc
    os.makedirs(destdir)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 7 more times]
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/root/.cache'

Looking in the issues, I found some related issues #55971, #55825 and #53664.

I do not understand the logic, I added some debug log in fileclient.py

--- /usr/lib/python3/dist-packages/salt/fileclient.py.orig  2020-03-04 15:45:28.742579156 +0100
+++ /usr/lib/python3/dist-packages/salt/fileclient.py   2020-03-04 15:45:11.598518180 +0100
@@ -151,6 +151,7 @@

             # ensure destdir exists
             try:
+                log.debug('DEBUG: destdir %s', destdir)
                 os.makedirs(destdir)
             except OSError as exc:
                 if exc.errno != errno.EEXIST:  # ignore if it was there already

I do not understand why the permission denied is on /root/.cache because the os.makedirs is called with a subdirectory:

[DEBUG   ] In saltenv 'base', looking at rel_path 'salt/pkgrepo/debian/install.sls' to resolve 'salt://salt/pkgrepo/debian/install.sls'
[DEBUG   ] DEBUG: destdir /home/dad/.salt/cache/files/base/salt/pkgrepo/debian
[DEBUG   ] In saltenv 'base', ** considering ** path '/home/dad/.salt/cache/files/base/salt/pkgrepo/debian/install.sls' to resolve 'salt://salt/pkgrepo/debian/install.sls'
[DEBUG   ] compile template: /home/dad/.salt/cache/files/base/salt/pkgrepo/debian/install.sls
[DEBUG   ] Jinja search path: ['/root/.cache/salt/thin/running_data/var/cache/salt/minion/files/base']
[DEBUG   ] In saltenv 'base', looking at rel_path 'salt/map.jinja' to resolve 'salt://salt/map.jinja'
[DEBUG   ] DEBUG: destdir /root/.cache/salt/thin/running_data/var/cache/salt/minion/files/base/salt
[DEBUG   ] Jinja Error
[DEBUG   ] Exception:
Traceback (most recent call last):
[...]
PermissionError: [Errno 13] Permission denied: '/root/.cache'

StefanSa commented 4 years ago

Seems to be the same problem, if so it is a serious bug, just tested on a CentOS and openSuSe 15.1. So uyuni cannot be used productively. Hopefully there will be a bugfix soon. https://github.com/uyuni-project/uyuni/issues/2437

JustOneMoreBlock commented 4 years ago

This was my solution:

mount -o remount,exec /tmp

mirko commented 3 years ago

I still experience this issue with v3002.1

mirko commented 3 years ago

It seems, that salt-ssh -- when called and targeting minions -- uses the same directory, as when targeting itself (localhost) as minion. Since I call salt-ssh as normal user (non-root) - /var/tmp/.conf_* is usually owned by that non-root user. When I however target the salt-master as minion (localhost), then salt-ssh uses the same directory for a) salt-master as well as b) salt-minion.

This clashes, when I run salt-ssh as a non-root user however have sudo: True specified in my roster for the salt-master host as a minion (typically localhost).

So, once I target thr salt-master host as salt-minion, the directory was created via sudo and hence can't be read/modified anymore by the non-root user salt-ssh is usually being called. Hence, all consecutive salt-ssh calls fail, because of wrong permissions.

zloyded commented 3 years ago

3002.2 same

An Exception occurred while executing state.show_top: [Errno 13] Permission denied: '/var/tmp/.root_7b0fba_salt/running_data'

pizzamig commented 3 years ago

The error is triggered only if I target the machine where the configuration is stored (local invocation)

salt-ssh is connecting to localhost via ssh as root user, and creates the thin_folder as root. Then, it tries to access the thin_folder as normal user and then fails with Permission denied. I tired to understand where salt is accessing the thin_folder directly (instead via ssh), but I failed to find the spot. My workaround is to recursively change the owner of the thin_folder (via chown -R) and re-run the salt-ssh command.

If I target a remote machine, I have no issue, it works just fine.

mirko commented 3 years ago

The error is triggered only if I target the machine where the configuration is stored (local invocation)

salt-ssh is connecting to localhost via ssh as root user, and creates the thin_folder as root. Then, it tries to access the thin_folder as normal user and then fails with Permission denied. I tired to understand where salt is accessing the thin_folder directly (instead via ssh), but I failed to find the spot. My workaround is to recursively change the owner of the thin_folder (via chown -R) and re-run the salt-ssh command.

If I target a remote machine, I have no issue, it works just fine.

This is what I was trying to describe in https://github.com/saltstack/salt/issues/46891#issuecomment-731771394 - it's still a very annoying bug, though :|

auphofBSF commented 3 years ago

I am having same issue version 3002.6 and agree with https://github.com/saltstack/salt/issues/46891#issuecomment-731771394 and https://github.com/saltstack/salt/issues/46891#issuecomment-749686915.

SUCCESFUL if first salt-ssh test.ping with roster sudo: false subsequent calls with salt-ssh can then be roster sudo: true or sudo: false

FAILS consistently if first salt-ssh call is with roster sudo: true and then subsequent call with roster sudo: false however sudo: true works.

I believe the /var/tmp/.<user>_<id>_salt directory should be created on first run with the chown as the roster user: <name>

anderbubble commented 3 years ago

I have this issue. At least, I think I have the issue that @MatthiasKuehneEllerhold originally reported. I am not convinced that all of the problem determination in this thread is the same issue.

Salt appears to construct Thin Salt in thin_dir. Salt also appears to use the eventual value for thin_dir on both the calling salt-ssh host and the called remote system. If you run salt-ssh as non-root, this means it puts non-root files (and directories) into the thin_dir path. But when the local and remote systems are the same system, you end up with root-owned files and directories in the accidentally-shared path that then cannot be updated in the local thin_dir, because they are root-owned.

I think the bug here is that Salt derives a single value for thin_dir and then uses it both locally and remotely. Particularly since the default value (/var/tmp/.%%USER%%_%%FQDNUUID%%_salt) has %%USER%% in it, it should be generated independently for each side of the salt-ssh communication. That way you would only have non-root files in the non-root thin_dir, and root files in the remote thin_dir.

I think the confusion has been that this looks like a remote error; but really it's an error on the local side. That's why, once it's broken, it's broken for all clients. The single attempt to run salt-ssh against the local node breaks salt-ssh itself. But if I remove /var/tmp/.*_salt and run only against other nodes it's fine. Once I've run against the local node, that run pollutes my thin_dir and it's broken "permanently."

anderbubble commented 2 years ago

fwiw, my use case is largely one of advocacy, or at least advocacy-motivated constraint. I'm trying to do all of my homelab Salt using salt-ssh so that I can use that experience to respond to my peers who use Ansible, and who claim that Ansible is easier to get started with because of its SSH-based agentless design. I want to be able to speak from experience that salt-ssh works just as well as Ansible's default SSH mechanism. That's why I specifically don't want to switch to salt-minion as a workaround, even for managing localhost: the first task an Ansible tutorial has you do is manage the local host, and if an Ansible person tries to replicate that experience with salt-ssh, the first thing they'll encounter is a pretty severe bug.

OrangeDog commented 2 years ago

salt-ssh works just as well as Ansible's default SSH mechanism

Unfortunately, it really doesn't. There are loads of SSH-specific bugs, none of which are being fixed.

I don't recommend using salt-ssh unless you really have to. Installing a local minion (including a masterless one) is always going to work better, and not send people rushing back to Ansible.

https://docs.saltproject.io/en/latest/topics/tutorials/quickstart.html

Salt is different to Ansible, especially if you're using it for configuration management. One of the reasons it will vastly out-perform Ansible is because it doesn't use master-driven SSH connections. It does provide built-in functionality to run Ansible playbooks to aid with migration. Expecting a starter Salt tutorial to be the same process as a starter Ansible tutorial doesn't make sense to me.

twangboy commented 1 year ago

Closing this issue due to age and lack of activity. Please test this on version 3006.2 and create a new issue if the problem persists. The new issue template has more information and will allow us to track and reproduce the issue more effectively. Thanks!

baby-gnu commented 1 year ago

I still have the issue so I open #65111

saltstack / salt