search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.23k stars 5.49k forks source link

Salt Client API timing out itself after running for ~5 hours[BUG] #61066

Open BhanupriyaH opened 3 years ago

BhanupriyaH commented 3 years ago

Description Have been using salt client API calls for the purpose of initiating a backup procedure on salt minions which takes several hours based on data present on hosts, we havent explicitly set any timeout for our cmd.run_all. However it is timing out by itself after approximately ~5.5 hours without giving dict type response even before the backup procedure is complete.

Setup Operating System: CentOS Linux 7 (Core) Kernel: Linux 3.10.0-1160.42.2.el7.x86_64 Salt Command timing out - LocalClient.cmd(target, cmd.run_all, "command to perform backup", tgt_type='list') target - list of minion-ids.

Steps to Reproduce the behavior 1) Run a backup/ long running job which takes more than 5 hours (Eg approximately backing up 100G of data) 2) Wait for SaltClient to return client return code

Issue noticed - salt client abruptly times out without waiting for job to complete and returns error without returning dictionary form of response. PS: Backup job initiated via salt is still running and not completed.

Expected behavior Salt client should wait until the job completes on minion and report back the status to salt master.

Screenshots Screenshot of salt client api return right before timeout. image

Versions Report

salt --versions-report Both minion and master are in same versions. ``` Salt Version: Salt: 3002.5 Dependency Versions: cffi: 1.14.5 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.11.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.14 pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Nov 16 2020, 16:55:22) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: centos 7 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1160.42.2.el7.x86_64 system: Linux version: CentOS Linux 7 Core ```

Additional context Tried even salt cli - salt "minion-id" cmd.run "command to perform backup", it failed with error: Minion did not return[No response] Not able to find any extra info even after enabling DEBUG logs. Cannot attach any details on debug logs as it is a remote box.

Any inputs appreciated. Thanks in advance.

welcome[bot] commented 3 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!