search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
13.98k stars 5.47k forks source link

[BUG] salt-master defunct after API calls using runner client #66566

Open jchatin opened 1 month ago

jchatin commented 1 month ago

Description salt-master defunct process after calling a runner function using the API with pepper cli #63838

Setup

Dockerfile ``` FROM debian:bullseye RUN apt-get update && apt-get upgrade -y && \ apt-get install -y wget gnupg2 RUN wget -O - https://repo.saltproject.io/salt/py3/debian/11/amd64/3006/SALT-PROJECT-GPG-PUBKEY-2023.pub | apt-key add - && \ printf 'deb https://repo.saltproject.io/salt/py3/debian/11/amd64/3006 bullseye main' > /etc/apt/sources.list.d/saltstack.list && \ apt-get update && \ apt-get install -y salt-master salt-api salt-ssh salt-cloud python3-pip git iproute2 && \ EXPOSE 4505 4506 8000 ADD run.sh /usr/local/bin/run.sh #ENV LOG_LEVEL debug #ENV LOG_LEVEL info RUN chmod +x "/usr/local/bin/run.sh" RUN useradd -p YourOwnPasswordHash testapi CMD [ "/usr/local/bin/run.sh" ] ```
/etc/salt/master.d/testapi.conf ``` rest_cherrypy: port: 80 host: 0.0.0.0 disable_ssl: true log_access_file: /var/log/salt/cherrypy_pap.access log_error_file: /var/log/salt/cherrypy_pap.error external_auth: pam: testapi: - .* - '@wheel' - '@runner' - '@jobs' ```

Steps to Reproduce the behavior install salt-pepper module and cli $ pip install salt-pepper

API call to salt master, to list jobs using the runner client $ pepper --client=runner -t 10 jobs.list_jobs

On salt master a defunct process salt-master is running. A new APi call will fork a new defunct.

Expected behavior No defunct process should run after runner jobs.list_jobs return

Screenshots

ps -ef --forest ``` root 4463 1 0 17:00 ? 00:00:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 6b8a025fd09e5c5f4dccbea907629650c97b25ed7ac9f2354f7d99945f27c651 -address /run/containerd/containerd.sock root 4483 4463 0 17:00 ? 00:00:04 \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error MainProcess root 4512 4483 0 17:00 ? 00:00:01 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-api --log-level=error MainProcess root 4544 4512 1 17:00 ? 00:00:44 | | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-api --log-level=error RunNetapi(salt.loaded.int.netapi.rest_cherrypy) root 4673 4483 0 17:00 ? 00:00:00 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error PubServerChannel._publish_daemon root 4675 4483 0 17:00 ? 00:00:00 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error EventPublisher root 4678 4483 0 17:00 ? 00:00:31 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error Maintenance root 4682 4483 5 17:00 ? 00:02:59 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error ConnectedCache root 19597 4682 1 17:59 ? 00:00:00 | | \_ [/opt/saltstack/] root 4708 4483 0 17:00 ? 00:00:00 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error ReqServer ReqServer_ProcessManager root 4709 4708 0 17:00 ? 00:00:00 | | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error ReqServer MWorkerQueue root 4710 4708 0 17:00 ? 00:00:04 | | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error ReqServer MWorker-0 root 4711 4708 0 17:00 ? 00:00:08 | | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error ReqServer MWorker-1 root 4715 4483 7 17:00 ? 00:04:31 | \_ /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error FileServerUpdate root 5226 4483 0 17:02 ? 00:00:00 | \_ [/opt/saltstack/] root 5227 4483 0 17:02 ? 00:00:01 | \_ [/opt/saltstack/] root 5274 4483 0 17:02 ? 00:00:00 | \_ [/opt/saltstack/] root 5275 4483 0 17:02 ? 00:00:01 | \_ [/opt/saltstack/] root 5323 4483 0 17:02 ? 00:00:00 | \_ [/opt/saltstack/] root 5324 4483 0 17:02 ? 00:00:01 | \_ [/opt/saltstack/] root 10240 4483 0 17:11 ? 00:00:00 | \_ [/opt/saltstack/] root 10241 4483 0 17:11 ? 00:00:01 | \_ [/opt/saltstack/] root 10420 4483 0 17:11 ? 00:00:00 | \_ [/opt/saltstack/] root 10421 4483 0 17:11 ? 00:00:01 | \_ [/opt/saltstack/] root 10601 4483 0 17:11 ? 00:00:00 | \_ [/opt/saltstack/] root 10602 4483 0 17:11 ? 00:00:01 | \_ [/opt/saltstack/] root 13147 4483 0 17:13 ? 00:00:00 | \_ [/opt/saltstack/] root 13148 4483 0 17:13 ? 00:00:01 | \_ [/opt/saltstack/] root 13233 4483 0 17:13 ? 00:00:00 | \_ [/opt/saltstack/] root 13236 4483 0 17:13 ? 00:00:01 | \_ [/opt/saltstack/] root 13318 4483 0 17:13 ? 00:00:00 | \_ [/opt/saltstack/] root 13319 4483 0 17:13 ? 00:00:01 | \_ [/opt/saltstack/] root 13570 4483 0 17:14 ? 00:00:00 | \_ [/opt/saltstack/] root 13571 4483 0 17:14 ? 00:00:01 | \_ [/opt/saltstack/] root 13613 4483 0 17:14 ? 00:00:00 | \_ [/opt/saltstack/] root 13614 4483 0 17:14 ? 00:00:01 | \_ [/opt/saltstack/] root 13655 4483 0 17:14 ? 00:00:00 | \_ [/opt/saltstack/] root 13657 4483 0 17:14 ? 00:00:01 | \_ [/opt/saltstack/] root 14591 4483 0 17:22 ? 00:00:00 | \_ [/opt/saltstack/] root 14592 4483 0 17:22 ? 00:00:01 | \_ [/opt/saltstack/] root 14640 4483 0 17:22 ? 00:00:00 | \_ [/opt/saltstack/] root 14642 4483 0 17:22 ? 00:00:01 | \_ [/opt/saltstack/] root 14954 4483 0 17:25 ? 00:00:00 | \_ [/opt/saltstack/] root 14955 4483 0 17:25 ? 00:00:01 | \_ [/opt/saltstack/] root 15011 4483 0 17:25 ? 00:00:00 | \_ [/opt/saltstack/] root 15012 4483 0 17:25 ? 00:00:01 | \_ [/opt/saltstack/] root 15090 4483 0 17:26 ? 00:00:00 | \_ [/opt/saltstack/] root 15091 4483 0 17:26 ? 00:00:01 | \_ [/opt/saltstack/] root 15154 4483 0 17:26 ? 00:00:00 | \_ [/opt/saltstack/] root 15155 4483 0 17:26 ? 00:00:01 | \_ [/opt/saltstack/] root 15286 4483 0 17:26 ? 00:00:00 | \_ [/opt/saltstack/] root 15287 4483 0 17:26 ? 00:00:01 | \_ [/opt/saltstack/] root 15346 4483 0 17:26 ? 00:00:00 | \_ [/opt/saltstack/] root 15347 4483 0 17:26 ? 00:00:01 | \_ [/opt/saltstack/] root 15399 4483 0 17:26 ? 00:00:00 | \_ [/opt/saltstack/] root 15400 4483 0 17:26 ? 00:00:01 | \_ [/opt/saltstack/] root 15497 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15498 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15557 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15558 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15601 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15602 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15647 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15649 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15709 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15710 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15751 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15752 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15809 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15810 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15853 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15854 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 15902 4483 0 17:27 ? 00:00:00 | \_ [/opt/saltstack/] root 15903 4483 0 17:27 ? 00:00:01 | \_ [/opt/saltstack/] root 16159 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16160 4483 0 17:29 ? 00:00:03 | \_ [/opt/saltstack/] root 16211 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16212 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] root 16266 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16267 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] root 16317 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16319 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] root 16362 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16363 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] root 16442 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16443 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] root 16491 4483 0 17:29 ? 00:00:00 | \_ [/opt/saltstack/] root 16492 4483 0 17:29 ? 00:00:01 | \_ [/opt/saltstack/] ```
master logfile ``` 2024-05-21 15:29:45,437 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master 2024-05-21 15:29:45,566 [salt.config :2180][DEBUG ][9363] Including configuration from '/etc/salt/master.d/testapi.conf' 2024-05-21 15:29:45,566 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master.d/testapi.conf 2024-05-21 15:29:45,574 [salt.loader :1073][DEBUG ][9363] Grains refresh requested. Refreshing grains. 2024-05-21 15:29:45,574 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master 2024-05-21 15:29:45,696 [salt.config :2180][DEBUG ][9363] Including configuration from '/etc/salt/master.d/testapi.conf' 2024-05-21 15:29:45,696 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master.d/testapi.conf 2024-05-21 15:29:45,728 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'core' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,730 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'disks' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,731 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'extra' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,734 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'lvm' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,735 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'mdadm' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,738 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'minion_process' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,740 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'opts' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,744 [salt.loader.lazy :876 ][DEBUG ][9363] Override __utils__: 2024-05-21 15:29:45,744 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'zfs' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,780 [salt.loaded.int.grains.core:2009][DEBUG ][9363] Could not determine init system from command line: (/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error MainProcess) 2024-05-21 15:29:45,797 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'zfs' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,798 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded zfs.is_supported 2024-05-21 15:29:45,880 [salt.utils.event :315 ][DEBUG ][9363] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc 2024-05-21 15:29:45,880 [salt.utils.event :316 ][DEBUG ][9363] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc 2024-05-21 15:29:45,906 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'jobs' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,907 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded jobs.list_jobs 2024-05-21 15:29:45,910 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'auth' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,912 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'cache' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,913 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'cloud' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,914 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'config' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,917 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'doc' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,920 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'error' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,920 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'event' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,923 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'fileserver' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,924 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'git_pillar' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,924 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'http' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,925 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'launchd' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,927 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'lxc' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,928 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'manage' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,930 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'mattermost' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,931 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'mine' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,935 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'network' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,936 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'pagerduty' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,937 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'pillar' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,938 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'pkg' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,939 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'queue' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,940 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'reactor' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,942 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'salt' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,943 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'saltutil' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,944 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'sdb' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,945 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'vmadm' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,946 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'ssh' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,947 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'state' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,948 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'survey' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,948 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'test' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,949 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'thin' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,950 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'vault' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,953 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'virt' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,954 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'winrepo' are being loaded by dir() on the loaded module 2024-05-21 15:29:45,960 [salt.utils.event :823 ][DEBUG ][9363] Sending event: tag = salt/run/20240521152945380149/new; data = {'fun': 'runner.jobs.list_jobs', 'jid': '20240521152945380149', 'user': 'testapi', 'fun_args': [], '_stamp': '2024-05-21T15:29:45.959960'} 2024-05-21 15:29:45,962 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master 2024-05-21 15:29:46,100 [salt.config :2180][DEBUG ][9363] Including configuration from '/etc/salt/master.d/testapi.conf' 2024-05-21 15:29:46,100 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master.d/testapi.conf 2024-05-21 15:29:46,106 [salt.loader :1073][DEBUG ][9363] Grains refresh requested. Refreshing grains. 2024-05-21 15:29:46,106 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master 2024-05-21 15:29:46,246 [salt.config :2180][DEBUG ][9363] Including configuration from '/etc/salt/master.d/testapi.conf' 2024-05-21 15:29:46,247 [salt.config :2017][DEBUG ][9363] Reading configuration from /etc/salt/master.d/testapi.conf 2024-05-21 15:29:46,278 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'core' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,280 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'disks' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,281 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'extra' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,283 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'lvm' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,284 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'mdadm' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,286 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'minion_process' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,288 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'opts' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,291 [salt.loader.lazy :876 ][DEBUG ][9363] Override __utils__: 2024-05-21 15:29:46,291 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'zfs' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,331 [salt.loaded.int.grains.core:2009][DEBUG ][9363] Could not determine init system from command line: (/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master --log-level=error MainProcess) 2024-05-21 15:29:46,349 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'zfs' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,349 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded zfs.is_supported 2024-05-21 15:29:46,450 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'jinja' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,450 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded jinja.render 2024-05-21 15:29:46,451 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'yaml' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,452 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded yaml.render 2024-05-21 15:29:46,455 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'local_cache' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,456 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded local_cache.get_jids 2024-05-21 15:29:46,468 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'local_cache' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,468 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded local_cache.prep_jid 2024-05-21 15:29:46,472 [salt.loaded.int.returner.local_cache:237 ][DEBUG ][9363] Adding minions for job 20240521152945380149: [] 2024-05-21 15:29:46,474 [salt.utils.event :823 ][DEBUG ][9363] Sending event: tag = salt/run/20240521152945380149/ret; data = {'fun': 'runner.jobs.list_jobs', 'jid': '20240521152945380149', 'user': 'testapi', 'fun_args': [], '_stamp': '2024-05-21T15:29:46.473974', 'return': {'20240521151335439202': {'Function': 'cmd.run', 'Arguments': ['…………….. …} 2024-05-21 15:29:46,486 [salt.loader.lazy :977 ][DEBUG ][9363] The functions from module 'nested' are being loaded by dir() on the loaded module 2024-05-21 15:29:46,486 [salt.utils.lazy :99 ][DEBUG ][9363] LazyLoaded nested.output 2024-05-21 15:29:46,495 [salt.client.mixins:441 ][INFO ][9363] Runner completed: 20240521152945380149 2024-05-21 15:29:46,496 [salt.transport.ipc:372 ][DEBUG ][9363] Closing IPCMessageClient instance ```

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3006.4 Python Version: Python: 3.10.13 (main, Oct 4 2023, 21:54:22) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: 1.7.1 looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.9.8 pygit2: 1.13.3 python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.13.12 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: debian 11 bullseye locale: utf-8 machine: x86_64 release: 5.10.0-28-cloud-amd64 system: Linux version: Debian GNU/Linux 11 bullseye ```

Additional context

welcome[bot] commented 1 month ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

jchatin commented 1 month ago

updated salt to 3007.0, same issue

Dockerfile ``` FROM debian:bullseye RUN apt-get update && apt-get upgrade -y && \ apt-get install -y wget gnupg2 RUN wget -O - https://repo.saltproject.io/salt/py3/debian/11/amd64/3007/SALT-PROJECT-GPG-PUBKEY-2023.pub | apt-key add - && \ printf 'deb https://repo.saltproject.io/salt/py3/debian/11/amd64/3007 bullseye main' > /etc/apt/sources.list.d/saltstack.list && \ apt-get update && \ apt-get install -y salt-master salt-api salt-ssh salt-cloud python3-pip git iproute2 && \ apt-get install -y patchelf EXPOSE 4505 4506 8000 ADD run.sh /usr/local/bin/run.sh #ENV LOG_LEVEL debug #ENV LOG_LEVEL info RUN chmod +x "/usr/local/bin/run.sh" RUN useradd -p YourOwnPasswordHash testapi CMD [ "/usr/local/bin/run.sh" ] ```
salt --versions-report ``` Salt Version: Salt: 3007.0 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: unknown dateutil: 2.8.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: 1.8.1 looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: 1.15.0 python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.15.1 smmap: Not Installed timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: debian 11.9 bullseye locale: utf-8 machine: x86_64 release: 5.10.0-28-cloud-amd64 system: Linux version: Debian GNU/Linux 11.9 bullseye ```
jchatin commented 1 month ago

Also just noticed that running a runner module from the salt master in async has the same behavior

From salt master

loop 'salt-run jobs.list_jobs --async' #while true;do salt-run jobs.list_jobs --async;sleep 2;done [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165915059925 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165918440154 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165921779997 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165924996628 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165928173393 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165931430030 [WARNING ] Running in asynchronous mode. Results of this execution may be collected by attaching to the master event bus or by examining the master job cache, if configured. This execution is running under tag salt/run/20240521165934535769
in parallel from another terminal , loop 'ps -ef|grep defunct' to count defunct processes #while true ; do ps -ef --forest| grep defun| wc -l; sleep 2.5; done 40 41 42 44 45 47 48 50 52 53