Report ENOEXEC error as "command failed to execute"

[greenmoss]

Description of Issue

I set up salt-proxy to run fx2/racadm. However upon startup, salt-proxy logged error "command not found".

This was very confusing, because I was able to run racadm from the command line, using the same $PATH.

After some digging, I discovered racadm was a symlink to an executable file, but the executable was missing a #!/usr/bin/env bash shebang. Bash followed the symlinks and ran the executable, but salt-proxy would not. Once I added the shebang, salt-proxy was able to run racadm.

Proposed solutions

To reduce confusion for future users, perhaps it would make sense to either:

• Report "command failed to execute" as opposed to "command not found"
• Alternately, allow salt-proxy to execute in this case, since bash is able to do so

Setup

# grep -v ^# /etc/salt/proxy | grep -v ^$
master: my-master
pillar_roots:
  base:
    - /etc/salt/pillar
# cat /etc/salt/pillar/top.sls
base:
  my-host-idrac:
    - my-host-idrac
# cat /etc/salt/pillar/my-host-idrac.sls
proxy:
  proxytype: fx2
  host: my-host-idrac
  admin_username: my-admin
  passwords:
    - my-pass

Steps to Reproduce Issue

# strace -f -o /tmp/proxy.out salt-proxy --proxyid my-host-idrac -l debug

Trimmed output other than stack trace:

[CRITICAL] Failed to load grains defined in grain file fx2.fx2 in function <function fx2 at 0x7f7efcef7398>, error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 798, in grains
    ret = funcs[key](**kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 87, in fx2
    return _grains()
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 69, in _grains
    (username, password) = _find_credentials()
  File "/usr/lib/python2.7/dist-packages/salt/grains/fx2.py", line 47, in _find_credentials
    admin_password=pwd)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1097, in get_chassis_name
    admin_password=admin_password)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 1394, in bare_rac_cmd
    admin_password=admin_password)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dracr.py", line 128, in __execute_ret
    output_loglevel='quiet')
  File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/modules/cmdmod.py", line 669, in _run
    raise CommandExecutionError(msg)
CommandExecutionError: Unable to run command 'REDACTED' with the context '{u'timeout': None, u'with_communicate': True, u'shell': False, u'bg': False, u'stderr': -1, u'env': {'LC_NUMERIC': 'C', 'LC_CTYPE': 'C', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LOGNAME': 'root', 'USER': 'root', 'HOME': '/home/greenmoss', 'LC_PAPER': 'C', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', 'DISPLAY': 'localhost:11.0', 'LANG': 'en_US.UTF-8', 'TERM': 'screen', 'SHELL': '/bin/bash', 'LANGUAGE': 'C', 'LC_MEASUREMENT': 'C', 'SUDO_USER': 'greenmoss', 'LC_MONETARY': 'C', 'USERNAME': 'root', 'LC_IDENTIFICATION': 'C', 'LC_ADDRESS': 'C', 'SUDO_UID': '12345', '_': '/usr/bin/strace', 'SUDO_COMMAND': '/bin/bash', 'SUDO_GID': '1234', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'LC_MESSAGES': 'C', 'OLDPWD': '/etc/salt', 'LC_TELEPHONE': 'C', 'SHLVL': '1', 'PWD': '/opt/local/bin', 'LC_NAME': 'C', 'MAIL': '/var/mail/root', 'LC_TIME': 'C', 'LC_COLLATE': 'C'}, u'stdout': -1, u'close_fds': True, u'stdin': None, u'cwd': u'/home/greenmoss'}', reason: command not found

trimmed exit messages, now looking at relevant section from strace:

32535 execve("/usr/local/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/local/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/usr/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/sbin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)
32535 execve("/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOEXEC (Exec format error)
32535 execve("/snap/bin/racadm", ["racadm", "-r", "my-host-idrac", "-u", "my-admin", "-p", "my-pass", "getchassisname"], [/* 34 vars */]) = -1 ENOENT (No such file or directory)

Note the ENOEXEC in the middle there, even though I can run it from shell:

# racadm help

 help [subcommand]    -- display usage summary for a subcommand
 clearasrscreen       -- clear the last ASR (crash) screen
(trimmed)

Now the weird chain of links, etc:

# which racadm
/bin/racadm
# ls -l /bin/racadm
lrwxrwxrwx 1 root root 30 Nov 23  2016 /bin/racadm -> /opt/dell/srvadmin/sbin/racadm
# ls -l /opt/dell/srvadmin/sbin/racadm
lrwxrwxrwx 1 root root 21 Nov 23  2016 /opt/dell/srvadmin/sbin/racadm -> racadm-wrapper-idrac7
# cat /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7
#directly invoke idracadm7 - it will check for generation and invoke required binary

exec /opt/dell/srvadmin/bin/idracadm7 "$@"

Fix: edit /opt/dell/srvadmin/sbin/racadm-wrapper-idrac7, add shebang:

#!/usr/bin/env bash
#directly invoke idracadm7 - it will check for generation and invoke required binary

exec /opt/dell/srvadmin/bin/idracadm7 "$@"

Versions Report

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4

System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-31-generic
         system: Linux
        version: Ubuntu 14.04 trusty

[dwoz]

@greenmoss Thank you for reporting this.

@cro Any thoughts on the best way to address this issue?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.