Salt pam Authentication suddenly fails

[pratiktest]

Hi,

I am using salt to manage my virtual machines. Till now I was using salt version salt-master 2015.5.3 (Lithium). I am also using salt-api to make remote calls (salt-api 2015.5.3)

root@salt-master-9660:/home/prkale# salt-master --version
salt-master 2015.5.3 (Lithium)
root@salt-master-9660:/home/prkale# salt-api --version
salt-api 2015.5.3
root@salt-master-9660:/home/prkale# 

I am not having any problems with this version. We decided to move to the latest Nitrogen version. I am using below script to install necessary configurations in my master. The only authentication i want is pam with user saltdev.

Below script installs Nitrogen successfully with all configurations i want Also The master works perfectly for some calls and after some time the /login call fails

#!/bin/sh

# get salt-stack repo for latest release of salt-master
wget -O - https://repo.saltstack.com/apt/ubuntu/16.04/amd64/2017.7/SALTSTACK-GPG-KEY.pub | apt-key add -

echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/2017.7 xenial main" >> /etc/apt/sources.list.d/saltstack.list

apt-get update

# install cherrypy
pip install --upgrade pip
apt-get install python-pip -y
pip install cherrypy

# add a user salt-dev. This user will be used to authenticate from salt-master
useradd saltdev --home /home/saltdev --shell /bin/bash
echo saltdev:**** | chpasswd

# installing salt-master
apt-get install salt-master -y

# installing salt-api
apt-get install salt-api

# modify  /etc/salt/master to include cherrypy module
echo "rest_cherrypy:
  port: 8000
  debug: True
  host: 0.0.0.0
  disable_ssl: True

open_mode: True

external_auth:
  pam:
    saltdev:
      - .*
      - '@wheel'
      - '@runner'
      - '@jobs'" >> /etc/salt/master

#create salt state serving directory
mkdir /srv/salt

# provide haproxy state
echo "stopHaproxy:
     cmd.run:
         - name: service haproxy stop;
copyConfig:
     cmd.run:
         - name: mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bck
         - onlyif: test -f /etc/haproxy/haproxy.cfg
createFile:
     file.append:
         - name: /etc/haproxy/haproxy.cfg
         - text: ''
appendFile:
     file.blockreplace:
         - name: /etc/haproxy/haproxy.cfg
         - content: |
                 {{ pillar['configData'] | indent(32) }}
         - append_if_not_found: True
startHaproxy:
     cmd.run:
         - name: service haproxy start" >> /srv/salt/haproxy.sls

# restart services
systemctl restart salt-master
systemctl restart salt-api

echo "end" >> /var/log/ecgs-init.log

If you see below in matter of minutes the same call says unauthorized (check the date error) We did not have this issue in lithium and for 2 years salt has been working without any issue. But this is preventing us from upgrading to nitrogen

prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='****' -d eauth='pam'
HTTP/1.1 200 OK
Content-Length: 205
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:11 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 6867a6a968af8e6c9e7245fe489e847f3abe0976
Content-Type: application/json
Set-Cookie: session_id=6867a6a968af8e6c9e7245fe489e847f3abe0976; expires=Thu, 15 Mar 2018 05:19:11 GMT; Max-Age=36000; Path=/

{"return": [{"perms": [".*", "@wheel", "@runner", "@jobs"], "start": 1521055151.13578, "token": "6867a6a968af8e6c9e7245fe489e847f3abe0976", "expire": 1521098351.135782, "user": "saltdev", "eauth": "pam"}]}prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='****' -d eauth='pam'
HTTP/1.1 200 OK
Content-Length: 206
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:14 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 48e55388450b2ef88aed4db88a0df62a547f0e9c
Content-Type: application/json
Set-Cookie: session_id=48e55388450b2ef88aed4db88a0df62a547f0e9c; expires=Thu, 15 Mar 2018 05:19:14 GMT; Max-Age=36000; Path=/

{"return": [{"perms": [".*", "@wheel", "@runner", "@jobs"], "start": 1521055154.296167, "token": "48e55388450b2ef88aed4db88a0df62a547f0e9c", "expire": 1521098354.296169, "user": "saltdev", "eauth": "pam"}]}prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='****' -d eauth='pam'
HTTP/1.1 200 OK
Content-Length: 205
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:16 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: eb3b9c52a096d1e833e3e95939b5b7a98840cf71
Content-Type: application/json
Set-Cookie: session_id=eb3b9c52a096d1e833e3e95939b5b7a98840cf71; expires=Thu, 15 Mar 2018 05:19:16 GMT; Max-Age=36000; Path=/

{"return": [{"perms": [".*", "@wheel", "@runner", "@jobs"], "start": 1521055156.384367, "token": "eb3b9c52a096d1e833e3e95939b5b7a98840cf71", "expire": 1521098356.38437, "user": "saltdev", "eauth": "pam"}]}prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='saltdev' -d eauth='pam'
HTTP/1.1 200 OK
Content-Length: 206
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:17 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 61a8910ba2f4e60384cdedf4c6307d1edac0a083
Content-Type: application/json
Set-Cookie: session_id=61a8910ba2f4e60384cdedf4c6307d1edac0a083; expires=Thu, 15 Mar 2018 05:19:17 GMT; Max-Age=36000; Path=/

{"return": [{"perms": [".*", "@wheel", "@runner", "@jobs"], "start": 1521055157.945451, "token": "61a8910ba2f4e60384cdedf4c6307d1edac0a083", "expire": 1521098357.945453, "user": "saltdev", "eauth": "pam"}]}prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='****' -d eauth='pam'
HTTP/1.1 200 OK
Content-Length: 205
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:21 GMT
Access-Control-Allow-Origin: *
X-Auth-Token: 24b3aa3b6688af857270a1e91331bf86e7d6fcce
Content-Type: application/json
Set-Cookie: session_id=24b3aa3b6688af857270a1e91331bf86e7d6fcce; expires=Thu, 15 Mar 2018 05:19:21 GMT; Max-Age=36000; Path=/

{"return": [{"perms": [".*", "@wheel", "@runner", "@jobs"], "start": 1521055161.061268, "token": "24b3aa3b6688af857270a1e91331bf86e7d6fcce", "expire": 1521098361.06127, "user": "saltdev", "eauth": "pam"}]}prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ 
prkale@salt-master-2152812:~$ curl -sSik http://localhost:8000/login -H "Accept: application/json" -d username='saltdev' -d password='****' -d eauth='pam'
HTTP/1.1 401 Unauthorized
Content-Length: 1752
Access-Control-Expose-Headers: GET, POST
Vary: Accept-Encoding
Server: CherryPy/14.0.0
Allow: GET, HEAD, POST
Access-Control-Allow-Credentials: true
Date: Wed, 14 Mar 2018 19:19:23 GMT
Access-Control-Allow-Origin: *
Content-Type: text/html;charset=utf-8
Set-Cookie: session_id=cefd0a6aeee8772bdf00a36a21e57c5569ecc57e; expires=Thu, 15 Mar 2018 05:19:23 GMT; Max-Age=36000; Path=/

<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>401 Unauthorized</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>401 Unauthorized</h2>
        <p>Could not authenticate using provided credentials</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 631, in respond
    self._do_respond(path_info)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 690, in _do_respond
    response.body = self.handler()
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 221, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/netapi/rest_cherrypy/app.py", line 755, in hypermedia_handler
    ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/netapi/rest_cherrypy/app.py", line 1771, in POST
    'Could not authenticate using provided credentials')
HTTPError: (401, 'Could not authenticate using provided credentials')
</pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 14.0.0</a>
      </span>
    </div>
    </body>
</html>

I also tried this salt command and see this error

root@salt-master-2152812:/var/log/salt# salt -a "pam" * test.ping
username: saltdev
password: 
Failed to authenticate! This is most likely because this user is not permitted to execute commands, but there is a small possibility that a disk error occurred (check disk/inode usage).

This is the master log

2018-03-14 12:13:38,861 [salt.utils.parsers:1048][WARNING ][4611] Master received a SIGTERM. Exiting.
2018-03-14 12:19:27,136 [salt.auth        :299 ][WARNING ][5887] Authentication failure of type "eauth" occurred.
2018-03-14 12:19:27,137 [salt.master      :1848][WARNING ][5887] Authentication failure of type "eauth" occurred.
2018-03-14 12:19:40,518 [salt.auth        :299 ][WARNING ][5879] Authentication failure of type "eauth" occurred.
2018-03-14 12:19:40,519 [salt.master      :1848][WARNING ][5879] Authentication failure of type "eauth" occurred.
2018-03-14 14:26:38,620 [salt.auth        :299 ][WARNING ][5890] Authentication failure of type "eauth" occurred.
2018-03-14 14:26:38,622 [salt.master      :1848][WARNING ][5890] Authentication failure of type "eauth" occurred.
2018-03-14 14:27:13,003 [salt.auth        :299 ][WARNING ][5892] Authentication failure of type "eauth" occurred.
2018-03-14 14:27:13,003 [salt.master      :1848][WARNING ][5892] Authentication failure of type "eauth" occurred.
2018-03-14 14:28:23,196 [salt.auth        :299 ][WARNING ][5887] Authentication failure of type "eauth" occurred.
2018-03-14 14:28:23,197 [salt.master      :1848][WARNING ][5887] Authentication failure of type "eauth" occurred.
2018-03-14 14:29:42,748 [salt.auth        :299 ][WARNING ][5879] Authentication failure of type "eauth" occurred.
2018-03-14 14:29:42,749 [salt.master      :1848][WARNING ][5879] Authentication failure of type "eauth" occurred.
2018-03-14 14:30:25,675 [salt.auth        :299 ][WARNING ][5886] Authentication failure of type "eauth" occurred.
2018-03-14 14:30:53,323 [salt.auth        :299 ][WARNING ][5890] Authentication failure of type "eauth" occurred.

[pratiktest]

This is happening in Ubuntu 16.04. I installed nitrogen in centoOS and the pam works fine

[garethgreenaway]

@pratiktest Thanks for the report. Can you check the version of CherryPy that are installed on both the Ubuntu 16.04 installation where you're seeing issues and the CentOS where you are not? Thanks!

[pratiktest]

oh the centOS version is different although i installed it using pip in both ubuntu and centOS

[root@salt-master-2154520 ~]# python -c "import cherrypy;print cherrypy.__version__"
5.6.0
[root@salt-master-2154520 ~]# 

I decommissioned the machine for ubuntu but i remember the version installed by pip for cherryPy was 14.0

This is how i installed cherrypy in centOS

easy_install pip
pip install cherrypy -y

however not sure if its a cherrypy issue since salt -a "pam" * test.ping too fails to authenticate

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.