[BUG] Migrating salt-master, cant reconnect minions

[rj-dsl]

Description After migrating my master from an Ubuntu 16.04 to a CentOS 8 VM, my master now cant initilize its key

Setup Standard setup,

Steps to Reproduce the behavior Rsynced etc/salt/master/*and /var/cache/salt/* to the new server, shutdown old server, started new server on old servers IP (for DNS resolving)

Expected behavior A clear and concise description of what you expected to happen.

Screenshots After minion restart I get the following on all minions:

May 08 22:31:10 MINION salt-minion[32144]: [ERROR   ] Error while bringing up minion for multi-master. Is master at IP responding?
May 08 22:31:30 MINION salt-minion[32144]: [ERROR   ] Minion unable to successfully connect to a Salt Master.
May 08 22:32:00 MINION salt-minion[32144]: [ERROR   ] Error while bringing up minion for multi-master. Is master at IP responding?
May 08 22:32:50 MINION salt-minion[32144]: [ERROR   ] Error while bringing up minion for multi-master. Is master at IP responding?

On the new master I am seeing the following:

May  7 16:29:18 localhost salt-master[987]: [ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7fd7801755f8> exception was never retrieved: Traceback (most recent call last):
May  7 16:29:18 localhost salt-master[987]:  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
May  7 16:29:18 localhost salt-master[987]:    yielded = next(result)
May  7 16:29:18 localhost salt-master[987]:  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 766, in handle_message
May  7 16:29:18 localhost salt-master[987]:    stream.send(self.serial.dumps(self._auth(payload['load'])))
May  7 16:29:18 localhost salt-master[987]:  File "/usr/lib/python3.6/site-packages/salt/transport/mixins/auth.py", line 513, in _auth
May  7 16:29:18 localhost salt-master[987]:    ret['sig'] = salt.crypt.private_encrypt(self.master_key.key, digest)
May  7 16:29:18 localhost salt-master[987]:  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 294, in private_encrypt
May  7 16:29:18 localhost salt-master[987]:    return key.private_encrypt(message, salt.utils.rsax931.RSA_X931_PADDING)
May  7 16:29:18 localhost salt-master[987]:  File "/usr/lib64/python3.6/site-packages/M2Crypto/RSA.py", line 76, in private_encrypt
May  7 16:29:18 localhost salt-master[987]:    assert self.check_key(), 'key is not initialised'
May  7 16:29:18 localhost salt-master[987]: AssertionError: key is not initialised

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) New master: ``` Salt Version: Salt: 3000.2 Dependency Versions: cffi: 1.11.5 cherrypy: Not Installed dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: 0.33.0 Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.6.1 mysql-python: Not Installed pycparser: 2.14 pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, May 21 2019, 23:51:36) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 17.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.1 System Versions: dist: centos 8.0.1905 Core locale: UTF-8 machine: x86_64 release: 4.18.0-80.el8.x86_64 system: Linux version: CentOS Linux 8.0.1905 Core ``` Minions: (all are on the same version and python3): ``` Salt Version: Salt: 3000.2 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.4.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.8 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.6.2 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 3.5.2 (default, Apr 16 2020, 17:47:17) python-gnupg: 0.3.8 PyYAML: 3.11 PyZMQ: 15.2.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: Ubuntu 16.04 xenial locale: UTF-8 machine: x86_64 release: 4.4.0-178-generic system: Linux version: Ubuntu 16.04 xenial ```

Additional context Googling gave me nothing, only post was This

[DmitryKuzmenko]

@rj-dsl thank you for report. Could you please clarify what is the salt-versions on your old master?

[rj-dsl]

Old master salt --versions-report :

Salt Version:
           Salt: 3000.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 (default, Apr 16 2020, 17:47:17)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-178-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[b0r0]

I have same issue on CentOS 7.8.2003. Py3 3000.3 version of salt-master have problem with M2Crypto function check_key, m2.rsa_check_key(self.rsa) throw an exception. If I changed salt-master 3000.3 to Py2 based everything is working fine. The check_key function is validating RSA keys and check that p and q are in fact prime. My master keys were generated near 2013 so I think, that there is problem with key generator. If I generate new keys for py3 salt-master everything is working fine (but require master key change on minions).

[rj-dsl]

Mine are from around 2012, so I guess its the same for me.. Any takes on how to fix this issue? Reverting to python2 is not really a viable option