tjyang commented 6 months ago

Description I downgraded to 3006.8(salt01) from 3007 and notice a simple test.ping take extra 4 seconds compared to 3007 master(rock01).

Please label it as Enhancement.

Setup time sudo salt -N testsalt test.ping

Please be as specific as possible and give set-up details.

[ x] on-prem machine
[x ] VM (Virtualbox, KVM, etc. please specify)
[ ] VM running on a cloud service, please be explicit and add details
[ ] container (Kubernetes, Docker, containerd, etc. please specify)
[ ] or a combination, please be explicit
[ ] jails if it is FreeBSD
[ ] classic packaging
[ x] onedir packaging
[ x] used bootstrap to install

Steps to Reproduce the behavior

[me@salt01 ~]$ time sudo salt -N testsalt test.ping # 15 minions in testsalt nodegroup.

real 0m5.686s user 0m0.648s sys 0m0.057s [me@salt01 ~]$ time sudo salt \* test.ping **Expected behavior** rock01 and salt01 masters are on same subnet. [me@rock01 ~]$ time sudo salt -N testsalt test.ping # same 15 minions in rocky01 real 0m1.033s user 0m0.017s sys 0m0.009s [me@rock01 ~]$ **Screenshots** If applicable, add screenshots to help explain your problem. **Versions Report**

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3006.8 Python Version: Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.16.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: rocky 8.9 Green Obsidian locale: utf-8 machine: x86_64 release: 4.18.0-513.24.1.el8_9.x86_64 system: Linux version: Rocky Linux 8.9 Green Obsidian ```

**Additional context** Add any other context about the problem here.

tjyang commented 5 months ago

Following test.ping time is from 3007.1 master and minions. lower than 1 second. But this bug: https://github.com/saltstack/salt/issues/66519 is stopping it as production ready.

-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 15
# of minions returned: 15
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------

real    0m0.967s
user    0m0.015s
sys     0m0.010s

tjyang commented 5 months ago

@dwoz and @Akm0d , Thanks for accepting this issue as bug and scheduled it to be fixed in 3006.9. Please let me know if you need further information from me.

Akm0d commented 5 months ago

I am unable to replicate the issue using a pure python install from git.

Akm0d commented 5 months ago

@tjyang If you reboot the machine then does the issue persist? Knowing this will help me pinpoint the root cause.

tjyang commented 5 months ago

@Akm0d

salt02 and salt01 are in multi-master both on 3006.8, I rebooted salt02 and do the test.ping on testsalt nodegroup with has 16 minions, see the masked test result below. let me know if you need more info.

[me@salt02 ~]$ sudo systemctl status salt-master  |grep Active
   Active: active (running) since Tue 2024-06-18 17:11:58 CDT; 1min 7s ago
[me@salt02 ~]$ time sudo salt -N testsalt test.ping |tail -20
<snipped>

-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 16
# of minions returned: 16
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------

real    0m5.830s
user    0m0.736s
sys     0m0.090s
[me@salt02 ~]$

tjyang commented 4 months ago

Same time spent for pinging its own minion. This is on RockyLinux using onedir rpm approach.
```
[me@salt02 ~]$ time sudo salt salt02 test.ping
salt02:
True
```

Summary

of minions targeted: 1

of minions returned: 1

of minions that did not return: 0

of minions with errors: 0

real 0m5.824s user 0m0.734s sys 0m0.080s [me@salt02 ~]$

Akm0d commented 4 months ago

I am still having a hard time replicating the issue. Can you show me the output of "salt salt02 test.ping --log-level=trace"? Also, give me a versions report for the master and minions.

tjyang commented 4 months ago

log at trace level, test ping saltmaster's own minion take 5+ secs.

[me@salt02 ~]$ time sudo salt --log-level=trace  salt02  test.ping
[TRACE   ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'debug', 'log_format': '%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'root'}
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE   ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[TRACE   ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_server.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/node-groups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/node-groups.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/syndic.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/syndic.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/top.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/top.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt02
[DEBUG   ] Missing configuration file: /root/.saltrc
[TRACE   ] Processing <bound method ExtendedTargetOptionsMixIn.process_pillar_target of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f84278a3d90>, <CustomOption at 0x7f8414b09ae0: -H/--hosts>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8414b1e440>, <CustomOption at 0x7f8414b09f60: -E/--pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03370>, <CustomOption at 0x7f8414b09fc0: -L/--list>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03400>, <CustomOption at 0x7f8414b0a110: -G/--grain>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03490>, <CustomOption at 0x7f8414b0a1a0: -P/--grain-pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03520>, <CustomOption at 0x7f8414b0a1d0: -N/--nodegroup>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b035b0>, <CustomOption at 0x7f8414b0a230: -R/--range>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b036d0>, <CustomOption at 0x7f8414b0a3b0: -C/--compound>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03760>, <CustomOption at 0x7f8412ae8bb0: -J/--pillar-pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f8412b03640>, <CustomOption at 0x7f8412ae8be0: -S/--ipcidr>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03880>, <CustomOption at 0x7f8412ae8fa0: --out-indent/--output-indent>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03910>, <CustomOption at 0x7f8412ae9000: --out-file-append/--output-file-append>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b037f0>, <CustomOption at 0x7f8412ae9030: --no-color/--no-colour>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b039a0>, <CustomOption at 0x7f8412ae9060: --force-color/--force-colour>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f8412b03a30>, <CustomOption at 0x7f8412ae9090: --state-output/--state_output>)
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_level of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_file of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_level_logfile of <salt.cli.salt.SaltCMD object at 0x7f841922cb80>>
[TRACE   ] Processing <function TargetOptionsMixIn._mixin_after_parsed at 0x7f8412b00700>
[TRACE   ] Processing <function OutputOptionsMixIn._mixin_after_parsed at 0x7f8412b00dc0>
[TRACE   ] Processing <function SaltCMDOptionParser._mixin_after_parsed at 0x7f8412b01900>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging_routines at 0x7f8412aef760>
[TRACE   ] Processing <function MergeConfigMixIn.__merge_config_with_cli at 0x7f8412aef1c0>
[TRACE   ] Processing <function LogLevelMixIn.__setup_console_logger_config at 0x7f8412aefa30>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logfile_logger_config at 0x7f8412aefac0>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging_config at 0x7f8412aefb50>
[TRACE   ] Processing <function LogLevelMixIn.__verify_logging at 0x7f8412aefc70>
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging at 0x7f8412aefbe0>
[TRACE   ] Setting up console logging: {'log_level': 'trace', 'log_format': '[%(levelname)-8s] %(message)s', 'date_format': '%H:%M:%S'}
[TRACE   ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'debug', 'log_format': '%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'root'}
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE   ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/file_server.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/file_server.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/node-groups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/node-groups.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/syndic.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/syndic.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/top.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/top.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt02
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc
[TRACE   ] ReqChannel send clear load={'cmd': 'publish', 'tgt': 'salt02', 'fun': 'test.ping', 'arg': [], 'key': '/9bvh28eiZFrNvlJAoWQNCiqekEYiPv+Sdken4cgY6boNdObDPqt77776ZyCI7YQXO5/PE6+H6A=', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'show_timeout': True, 'show_jid': False, 'delimiter': ':'}, 'user': 'sudo_tjyang', 'to': 5}
[DEBUG   ] Closing AsyncReqChannel instance
[TRACE   ] func get_cli_event_returns()
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha1' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'hmac_compute' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'ifelse' as a jinja global
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'indent' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Marking 'method_call' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[TRACE   ] loaded json json lib
[DEBUG   ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/48/438fa51f2a24bd8f44e542f84e3c938ed2e2685c6b7047914f6f44e7b97e74/.minions.p
[DEBUG   ] get_iter_returns for jid 20240620230015298088 sent to {'salt02'} will timeout at 18:00:20.471577
[TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE   ] get_event() received = {'data': {'jid': '20240620230015298088', 'tgt_type': 'glob', 'tgt': 'salt02', 'user': 'sudo_tjyang', 'fun': 'test.ping', 'arg': [], 'minions': ['salt02'], 'missing': [], '_stamp': '2024-06-20T23:00:15.300576'}, 'tag': 'salt/job/20240620230015298088/new'}
[TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
[TRACE   ] get_event() received = {'data': {'cmd': '_return', 'id': 'salt02', 'success': True, 'return': True, 'retcode': 0, 'jid': '20240620230015298088', 'fun': 'test.ping', 'fun_args': [], 'user': 'sudo_tjyang', '_stamp': '2024-06-20T23:00:15.421773'}, 'tag': 'salt/job/20240620230015298088/ret/salt02'}
[DEBUG   ] jid 20240620230015298088 return from salt02
[DEBUG   ] return event: {'salt02': {'ret': True, 'retcode': 0, 'jid': '20240620230015298088'}}
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {'salt02': True}
salt02:
    True
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [TRACE   ] Get event. tag: (salt/job|syndic/.*)/20240620230015298088
    [TRACE   ] _get_event() waited 0 seconds and received nothing
    [DEBUG   ] Closing IPCMessageSubscriber instance

-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 1
# of minions returned: 1
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------

real    0m5.839s
user    0m0.782s
sys     0m0.091s
[me@salt02 ~]$

Minion info: sudo salt-call -V

Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.9 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.24.1.el8_9.x86_64
        system: Linux
       version: Rocky Linux 8.9 Green Obsidian

Master version info: sudo salt -V

Salt Version:
          Salt: 3006.8

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.9 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.24.1.el8_9.x86_64
        system: Linux
       version: Rocky Linux 8.9 Green Obsidian

network-shark commented 4 months ago

This is my result

╰─ salt -V
Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:33:04) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: 1.5.0
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: 1.11.1
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 11.9 bullseye
        locale: utf-8
       machine: aarch64
       release: 5.10.103-v8+
        system: Linux
       version: Debian GNU/Linux 11.9 bullseye

sudo salt -N group4 test.ping  3.34s user 0.35s system 80% cpu 4.606 total

sudo salt testminion test.ping  2.44s user 0.27s system 86% cpu 3.134 total

tjyang commented 4 months ago

@network-shark, this issue is created for minion/master version 3006.8, not 3007.1.

tjyang commented 4 months ago

@Akm0d

Looks like I have misconfiguration somewhere in my production server to cause the delay. Please close this issue.

Another 3006.8 master/minion in my test lab running as VMWare VM.

test.ping itself is less than 1 second.


[me@rocky9t01 ~]$ time sudo salt rocky9t01 -l trace  test.version
[TRACE   ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE   ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[TRACE   ] Processing <bound method SaltfileMixIn.process_saltfile of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method ConfigDirMixIn.process_config_dir of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/acl.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/acl.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/returners.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/returners.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: rocky9t01
[DEBUG   ] Missing configuration file: /root/.saltrc
[TRACE   ] Processing <bound method ExtendedTargetOptionsMixIn.process_pillar_target of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_output of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_output_file of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method OutputOptionsMixIn.process_state_verbose of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method NoParseMixin.process_no_parse of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e30b63d90>, <CustomOption at 0x7f0e2ef45b10: -H/--hosts>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2ef5a440>, <CustomOption at 0x7f0e2ef45f90: -E/--pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f370>, <CustomOption at 0x7f0e2ef45ff0: -L/--list>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f400>, <CustomOption at 0x7f0e2ef46140: -G/--grain>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f490>, <CustomOption at 0x7f0e2ef461d0: -P/--grain-pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f520>, <CustomOption at 0x7f0e2ef46200: -N/--nodegroup>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f5b0>, <CustomOption at 0x7f0e2ef46260: -R/--range>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f6d0>, <CustomOption at 0x7f0e2ef463e0: -C/--compound>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f760>, <CustomOption at 0x7f0e2d754be0: -J/--pillar-pcre>)
[TRACE   ] Processing functools.partial(<function TargetOptionsMixIn._create_process_functions.<locals>.process at 0x7f0e2d76f640>, <CustomOption at 0x7f0e2d754c10: -S/--ipcidr>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f880>, <CustomOption at 0x7f0e2d754fd0: --out-indent/--output-indent>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f910>, <CustomOption at 0x7f0e2d755030: --out-file-append/--output-file-append>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f7f0>, <CustomOption at 0x7f0e2d755060: --no-color/--no-colour>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76f9a0>, <CustomOption at 0x7f0e2d755090: --force-color/--force-colour>)
[TRACE   ] Processing functools.partial(<function OutputOptionsMixIn._mixin_setup.<locals>.process at 0x7f0e2d76fa30>, <CustomOption at 0x7f0e2d7550c0: --state-output/--state_output>)
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_level of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_file of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <bound method LogLevelMixIn.process_log_level_logfile of <salt.cli.salt.SaltCMD object at 0x7f0e30720bb0>>
[TRACE   ] Processing <function TargetOptionsMixIn._mixin_after_parsed at 0x7f0e2d76c700>
[TRACE   ] Processing <function OutputOptionsMixIn._mixin_after_parsed at 0x7f0e2d76cdc0>
[TRACE   ] Processing <function SaltCMDOptionParser._mixin_after_parsed at 0x7f0e2d76d900>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging_routines at 0x7f0e2d75b760>
[TRACE   ] Processing <function MergeConfigMixIn.__merge_config_with_cli at 0x7f0e2d75b1c0>
[TRACE   ] Processing <function LogLevelMixIn.__setup_console_logger_config at 0x7f0e2d75ba30>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logfile_logger_config at 0x7f0e2d75bac0>
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging_config at 0x7f0e2d75bb50>
[TRACE   ] Processing <function LogLevelMixIn.__verify_logging at 0x7f0e2d75bc70>
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[TRACE   ] Processing <function LogLevelMixIn.__setup_logging at 0x7f0e2d75bbe0>
[TRACE   ] Setting up console logging: {'log_level': 'trace', 'log_format': '[%(levelname)-8s] %(message)s', 'date_format': '%H:%M:%S'}
[TRACE   ] Setting up log file logging: {'log_path': '/var/log/salt/master', 'log_level': 'warning', 'log_format': '%(asctime)s,%(msecs)03d [%(name)-17s:%(lineno)-4d][%(levelname)-8s][%(process)d] %(message)s', 'date_format': '%Y-%m-%d %H:%M:%S', 'max_bytes': 0, 'backup_count': 0, 'user': 'salt'}
[TRACE   ] The required configuration section, 'fluent_handler', was not found the in the configuration. Not loading the fluent logging handlers module.
[TRACE   ] None of the required configuration sections, 'logstash_udp_handler' and 'logstash_zmq_handler', were found in the configuration. Not loading the Logstash logging handlers module.
[TRACE   ] Error loading log_handlers.sentry_mod: Cannot find 'raven' python library, 'sentry_handler' config is empty or not defined
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/acl.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/acl.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/autosign_grains.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/autosign_grains.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/returners.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/returners.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/salt-api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/salt-api.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: rocky9t01
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pub.ipc
[TRACE   ] ReqChannel send clear load={'cmd': 'publish', 'tgt': 'rocky9t01', 'fun': 'test.version', 'arg': [], 'key': 'B5sLpYwuKvNAj56LPptQMGChQ3c+9yBEkmhXvkx01oHOfE2DHp7jw/e55wzLC91hnZDPO3Yn/F4=', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'show_timeout': True, 'show_jid': False, 'delimiter': ':'}, 'user': 'sudo_me'}
[DEBUG   ] Closing AsyncReqChannel instance
[TRACE   ] func get_cli_event_returns()
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha1' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'hmac_compute' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'ifelse' as a jinja global
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'indent' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Marking 'method_call' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[TRACE   ] loaded json json lib
[DEBUG   ] The functions from module 'local_cache' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/93/48d208fe005cd60e7e1faddbcd1a76f44203a8ccd383213e630d6b5cc93c36/.minions.p
[DEBUG   ] get_iter_returns for jid 20240624193307945925 sent to {'rocky9t01'} will timeout at 14:33:13.092168
[TRACE   ] Get event. tag: salt/job/20240624193307945925
[TRACE   ] _get_event() waited 0 seconds and received nothing
[TRACE   ] Get event. tag: salt/job/20240624193307945925
[TRACE   ] get_event() received = {'data': {'jid': '20240624193307945925', 'tgt_type': 'glob', 'tgt': 'rocky9t01', 'user': 'sudo_me', 'fun': 'test.version', 'arg': [], 'minions': ['rocky9t01'], 'missing': [], '_stamp': '2024-06-24T19:33:07.947187'}, 'tag': 'salt/job/20240624193307945925/new'}
[TRACE   ] Get event. tag: salt/job/20240624193307945925
[TRACE   ] get_event() received = {'data': {'cmd': '_return', 'id': 'rocky9t01', 'success': True, 'return': '3006.8', 'retcode': 0, 'jid': '20240624193307945925', 'fun': 'test.version', 'fun_args': [], 'user': 'sudo_me', '_stamp': '2024-06-24T19:33:08.054649'}, 'tag': 'salt/job/20240624193307945925/ret/rocky9t01'}
[DEBUG   ] jid 20240624193307945925 return from rocky9t01
[DEBUG   ] return event: {'rocky9t01': {'ret': '3006.8', 'retcode': 0, 'jid': '20240624193307945925'}}
[DEBUG   ] The functions from module 'nested' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {'rocky9t01': '3006.8'}
rocky9t01:
3006.8
[TRACE   ] Get event. tag: salt/job/20240624193307945925
[TRACE   ] _get_event() waited 0 seconds and received nothing
[DEBUG   ] jid 20240624193307945925 found all minions {'rocky9t01'}
[DEBUG   ] Closing IPCMessageSubscriber instance

real 0m0.834s user 0m0.662s sys 0m0.052s [me@rocky9t01 ~]$

- minion: salt-call -V

[me@rocky9t01 ~]$ salt-call -V Salt Version: Salt: 3006.8

Python Version: Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.16.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rocky 9.4 Blue Onyx locale: utf-8 machine: x86_64 release: 5.14.0-427.22.1.el9_4.x86_64 system: Linux version: Rocky Linux 9.4 Blue Onyx

[me@rocky9t01 ~]$

- master: salt -V

[me@rocky9t01 ~]$ salt -V Salt Version: Salt: 3006.8

Python Version: Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.16.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rocky 9.4 Blue Onyx locale: utf-8 machine: x86_64 release: 5.14.0-427.22.1.el9_4.x86_64 system: Linux version: Rocky Linux 9.4 Blue Onyx

[me@rocky9t01 ~]$

tjyang commented 3 months ago

This is a sub one sec test.ping on rockylinux 8 test master instance.
- Still need to figure out what caused the production master instances spent extra 4 secs.
```
[me@rocky8t01 ~]$ time sudo salt rocky8t01  test.ping
rocky8t01:
True
```

Summary

of minions targeted: 1

of minions returned: 1

of minions that did not return: 0

of minions with errors: 0

real 0m0.998s user 0m0.013s sys 0m0.013s [me@rocky8t01 ~]$ cat /etc/redhat-release Rocky Linux release 8.10 (Green Obsidian) [me@rocky8t01 ~]$ salt-call -V Salt Version: Salt: 3006.8

Python Version: Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions: cffi: 1.14.6 cherrypy: 18.6.1 dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.4.8 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.16.0 smmap: Not Installed timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4

System Versions: dist: rocky 8.10 Green Obsidian locale: utf-8 machine: x86_64 release: 4.18.0-553.el8_10.x86_64 system: Linux version: Rocky Linux 8.10 Green Obsidian

[me@rocky8t01 ~]$ Wed Jul 31 07:28:48 EDT 2024

tjyang commented 4 weeks ago

@Akm0d and all

The extra 4 seconds delay is found and fixed by comment out order_masters: True line.

This is my syndic.conf in /etc/salt/master.d , master was set True but syndic_master was set to fake name.


#####          Syndic settings       #####
##########################################
# The Salt syndic is used to pass commands through a master from a higher
# master. Using the syndic is simple. If this is a master that will have
# syndic servers(s) below it, then set the "order_masters" setting to True.
#
# If this is a master that will be running a syndic daemon for passthrough, then
# the "syndic_master" setting needs to be set to the location of the master server
# to receive commands from.

saltstack / salt

[BUG] 3006.8 master take extra 4 seconds to finish test.ping #66530

Summary

of minions targeted: 1

of minions returned: 1

of minions that did not return: 0

of minions with errors: 0

Summary

of minions targeted: 1

of minions returned: 1

of minions that did not return: 0

of minions with errors: 0

Set the order_masters setting to True if this master will command lower

masters' syndic interfaces.

order_masters: False

order_masters: True

If this master will be running a salt syndic daemon, syndic_master tells

this master where to receive commands from.

syndic_master: masterofmaster