mruepp
commented
6 years ago Output on the master:
[INFO ] Executing state salt.function for [pkg.upgrade]
[ERROR ] No changes made for pkg.upgrade ID: upgrade_all_packages
Function: salt.function
Name: pkg.upgrade
Result: False
Comment:
Started: 11:17:44.752874
Duration: 81940.991 ms
Changes:Event output:
"salt_|-upgrade_all_packages_|-pkg.upgrade_|-function": {
"__id__": "upgrade_all_packages",
"__run_num__": 4,
"__sls__": "orchestrate.prime",
"changes": {},
"command": "No minions responded",
"comment": "",
"duration": 81940.991,
"name": "pkg.upgrade",
"result": false,
"start_time": "11:17:44.752874"
},At the end, somewhat 2 minutes after the orchestrator returns failed, we see the update events with success true:
salt/job/20181017111744866510/ret/devs0242 {
"_stamp": "2018-10-17T09:21:20.817618",
"cmd": "_return",
"fun": "pkg.upgrade",
"fun_args": [],
"id": "devs0242",
"jid": "20181017111744866510",
"master_id": "salt.dev.sc.intra",
"retcode": 0,
"return": {
"NetworkManager": {
"new": "1:1.10.2-16.el7_5",
"old": "1:1.10.2-13.el7"
},
...
"tzdata": {
"new": "2018e-3.el7",
"old": "2018c-1.el7"
},
"util-linux": {
"new": "2.23.2-52.el7_5.1",
"old": "2.23.2-52.el7"
},
"vdo": {
"new": "6.1.0.168-18",
"old": "6.1.0.149-16"
},
"yum-plugin-fastestmirror": {
"new": "1.1.31-46.el7_5",
"old": "1.1.31-45.el7"
},
"yum-utils": {
"new": "1.1.31-46.el7_5",
"old": "1.1.31-45.el7"
}
},
"success": true
} 
gtmanfred
stale[bot]
Description of Issue/Question
We run an orchestrator which ultimately ends in a
The orchestrator runs fine but when we upgrade quite a bit of packages, it finishes with a failed state of the both. What really bothers is the upgrade state, because, when watching the events with:
salt-run state.event pretty=Truethe events continue after the return of the states failed, after some time the updates and the reboot happens, despite the states failed.
So somehow the ret will happen despite the state is not finished but returns failed despite it is still running the upgrade.
When running orchestration it is expected to run all the states in a defined order, and wait til the states finish for sure.
We tried increasing the
tried to apply a sleep state after the upgrade state, no success.
So why does the pkg.upgrade return failed to the event system despite it is not finished, and after all finish successfully, but returns failed?
Btw, we recognized also a lot of
Minion did not return. [No response]when applying the cmd on the salt master with:salt 'minion' pgk.upgradeas well. If you apply the state a second time, e.g. the process does not take so much time, it returns successfullyIts just an ordinary centos system, upgrading from 1803 to 1804. Somewhat about 250MB and about 3-5min when issuing
yum upgradeVersions Report
Versions Report
salt-minion --versions-report Salt Version: Salt: 2018.3.2
Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: 1.10.6 gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: Not Installed libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: Not Installed Python: 2.7.5 (default, Jul 13 2018, 13:06:57) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4
System Versions: dist: centos 7.5.1804 Core locale: UTF-8 machine: x86_64 release: 3.10.0-862.11.6.el7.x86_64 system: Linux version: CentOS Linux 7.5.1804 Core
salt --versions-report Salt Version: Salt: 2018.3.2
Dependency Versions: cffi: 1.6.0 cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed Jinja2: 2.7.2 libgit2: 0.26.3 libnacl: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack-pure: Not Installed msgpack-python: 0.5.6 mysql-python: Not Installed pycparser: 2.14 pycrypto: 2.6.1 pycryptodome: Not Installed pygit2: 0.26.4 Python: 2.7.5 (default, Jul 13 2018, 13:06:57) python-gnupg: Not Installed PyYAML: 3.11 PyZMQ: 15.3.0 RAET: Not Installed smmap: Not Installed timelib: Not Installed Tornado: 4.2.1 ZMQ: 4.1.4
System Versions: dist: centos 7.5.1804 Core locale: UTF-8 machine: x86_64 release: 3.10.0-862.14.4.el7.x86_64 system: Linux version: CentOS Linux 7.5.1804 Core