[BUG] salt-api is not working on salt 3007.1 (rocky/rhel9)

[sayf-eddine-scality]

Description Running salt-api with the following config

external_auth:
  pam:
    saltapi:
    - .*
    - '@runner'
netapi_enable_clients:
- local
rest_cherrypy:
  disable_ssl: true
  host: localhost
  port: 8001
log-level: debug

Fails with the following exception

[INFO    ] [31/Jul/2024:10:48:05] ENGINE Bus STARTED
[INFO    ] [api_acl] Authentication not checked for user saltapi from IP ::1
[DEBUG   ] Using selector: EpollSelector
[DEBUG   ] Error while processing request for: /login
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/salt/netapi/rest_cherrypy/app.py", line 860, in hypermedia_handler
    ret = cherrypy.serving.request._hypermedia_inner_handler(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/site-packages/salt/netapi/rest_cherrypy/app.py", line 1888, in POST
    token = self.auth.mk_token(creds)
  File "/usr/lib/python3.9/site-packages/salt/auth/__init__.py", line 594, in mk_token
    tdata = self._send_token_request(load)
  File "/usr/lib/python3.9/site-packages/salt/auth/__init__.py", line 523, in _send_token_request
    with salt.channel.client.ReqChannel.factory(
  File "/usr/lib/python3.9/site-packages/salt/channel/client.py", line 56, in factory
    return SyncWrapper(
  File "/usr/lib/python3.9/site-packages/salt/utils/asynchronous.py", line 76, in __init__
    self.obj = cls(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/salt/channel/client.py", line 139, in factory
    transport = salt.transport.request_client(opts, io_loop=io_loop)
  File "/usr/lib/python3.9/site-packages/salt/transport/base.py", line 59, in request_client
    return salt.transport.zeromq.RequestClient(opts, io_loop=io_loop)
  File "/usr/lib/python3.9/site-packages/salt/transport/zeromq.py", line 1085, in __init__
    self.sending = asyncio.Lock()
  File "/usr/lib64/python3.9/asyncio/locks.py", line 81, in __init__
    self._loop = events.get_event_loop()
  File "/usr/lib64/python3.9/asyncio/events.py", line 645, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread "('CP Server Thread-3',)".

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• [ ] on-prem machine
• [x] VM (Virtualbox, KVM, etc. please specify)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD
• [x] classic packaging
• [ ] onedir packaging
• [ ] used bootstrap to install

Steps to Reproduce the behavior (Include debug logs if possible and relevant)

Expected behavior Salt API to respond 200 OK

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3007.1 Python Version: Python: 3.9.18 (main, Jul 3 2024, 00:00:00) Dependency Versions: cffi: 1.14.5 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: Not Installed looseversion: 1.2.0 M2Crypto: 0.38.0 Mako: 1.1.2 msgpack: 1.0.3 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 21.3 pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.20.0 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 5.4.1 PyZMQ: 22.3.0 relenv: Not Installed smmap: Not Installed timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: Not Installed System Versions: dist: rhel 9.4 Plow locale: utf-8 machine: x86_64 release: 5.14.0-362.13.1.el9_3.x86_64 system: Linux version: Red Hat Enterprise Linux 9.4 Plow ```

Additional context It seems the ioloop is not sent to the Lock object, because when I modifie it manually vim /usr/lib/python3.9/site-packages/salt/transport/zeromq.py +1084

+        self.sending = asyncio.Lock(loop=self.io_loop)
-        self.sending = asyncio.Lock()

It works.

[tjyang]

Hi @sayf-eddine-scality I am interested to regain salt-api function on RockLinux/RHEL using onedir(self-contained python in /opt/salt/*). I will setup test VM for this endeavor.

[dmurphy18]

@sayf-eddine-scality From the version of Python

Python Version:
        Python: 3.9.18 (main, Jul  3 2024, 00:00:00)

I can assume that you used bootstrap to install Salt, rather than the packaged versions which use Python 3.10. The most recent version of the bootstrap script now installs salt-api, previously it did not.

I suggest that you use the packaged versions of Salt, see https://docs.saltproject.io/salt/install-guide/en/latest/topics/install-by-operating-system/rhel.html#install-salt-on-redhat-rhel-9-x86-64.

The only other way to install Salt and have Python 3.9, other than bootstrap would be to pip install salt, recommend installing packages which are tested and supported (recommend for production environments), otherwise your mileage may vary.

If this resolves your issue, please consider closing it, unless you have other information pertinent to the issue

[sayf-eddine-scality]

we actually package salt ourselves from the source, you think the issue is python3.9? I ll dig deeper

[dmurphy18]

@sayf-eddine-scality You do your own packaging then you are responsible for delivery of salt-api, and I shall close this as not the Salt Projects problem.

Yes, Python 3.9 is a problem since the onedir architecture embeds Python 3.10 and everything is tested using Python 3.10 (currently working on getting Python 3.12 working but having issues). Python 3.10 will have deprecated some items and may have removed them that are present in Python 3.9.

Given that there are perfectly fine packages for RHEL 9 available from Salt, and the reduced size of the Salt Team recently, not that much help can be fore-coming from the Salt Team on helping you create your own packages.