salt-minion 3004.1-1 on windows 10 : RSA key format is not supported

[baby-gnu]

Description

On some minions, the service can't start and the logs are filled with:

2022-06-03 07:03:58,699 [salt.minion :1166][CRITICAL][4704] Unexpected error while connecting to salt

This is not systematic.

Setup

This is a normal installation of the minion for windows without doing anything special.

• [X] on-prem machine
• [X] VM KVM
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD

Steps to Reproduce the behavior

1. install the minion for windows.

Expected behavior

The service should start correctly without any error in the logs

Screenshots

Here are some logs:

2022-06-03 07:03:58,699 [salt.minion      :1166][CRITICAL][4704] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\minion.py", line 1140, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\minion.py", line 1366, in connect_master
    master, self.pub_channel = yield self.eval_master(
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\ext\tornado\gen.py", line 309, in wrapper
    yielded = next(result)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\minion.py", line 821, in eval_master
    pub_channel = salt.transport.client.AsyncPubChannel.factory(
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\transport\client.py", line 205, in factory
    return salt.transport.zeromq.AsyncZeroMQPubChannel(opts, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\transport\zeromq.py", line 384, in __init__
    self.auth = salt.crypt.AsyncAuth(self.opts, io_loop=self.io_loop)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\crypt.py", line 515, in __new__
    auth.__singleton_init__(opts, io_loop=io_loop)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\crypt.py", line 551, in __singleton_init__
    self.get_keys()
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\crypt.py", line 903, in get_keys
    key = get_rsa_key(self.rsa_path, None)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\crypt.py", line 213, in get_rsa_key
    return _get_key_with_evict(path, str(os.path.getmtime(path)), passphrase)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\utils\decorators\__init__.py", line 298, in _memoize
    cache[args_] = func(*args, **kwargs)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\salt-3004.1-py3.8.egg\salt\crypt.py", line 197, in _get_key_with_evict
    key = RSA.importKey(f.read(), passphrase)
  File "C:\Program Files\Salt Project\Salt\bin\lib\site-packages\Cryptodome\PublicKey\RSA.py", line 788, in import_key
    raise ValueError("RSA key format is not supported")
ValueError: *RSA key format is not supported*

Versions Report

salt --versions-report

```yaml Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.7 gitpython: Not Installed Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: Not Installed Mako: 1.1.4 msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.10.1 pygit2: Not Installed Python: 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)] python-gnupg: 0.4.7 PyYAML: 5.4.1 PyZMQ: 19.0.0 smmap: 4.0.0 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: locale: cp1252 machine: AMD64 release: 10 system: Windows version: 10 10.0.19041 SP0 Multiprocessor Free ```

Additional context

I had the same issue on the master some time ago and found that it was caused by some checks in pycrtodome.

According to the source M2Crypto seems the prefered one and may be installed instead of pycrptodome, right?

Other reports were closed because it's hard to reproduce what cause the pycrotodome to reject the generated key:

• 40889

[twangboy]

This could be related: https://github.com/saltstack/salt/issues/61131 I would recommend you use the same crypto library on your master and minions. Since you're using Windows, I would recommend sticking with pycryptodome since m2crypto is tricky to install on Windows.

[baby-gnu]

That's quite unfortunate since m2crypto was the solution for the unreadable master.pem key.

So, in some situation it's better to have m2crypto and in other it should be pycryptodome.

The solution is not simple to avoid manual intervention on all faulty minions :thinking: