[BUG][Debian] updating minion from 3001 to 3002 makes minion unreachable

[Yoda-BZH]

Your setup

Formula commit hash / release tag

using pillarstack master + minion salt-formula: 1.5.1 / 2dcd0c80efbd0acd0ca832a3435b7f3dc21788eb

Versions reports (master & minion)

master: 3002+ds-1 syndic: 3001.3+ds-1 salt-formula: 1.5.1 / 2dcd0c80efbd0acd0ca832a3435b7f3dc21788eb Debian: 10

Pillar / config used

Using this pillarstack configuration:

states:
  list:
  - salt
  - salt.fixes

{% set masters = __opts__.get('default_masters', ['salt-master', ]) %}
salt:
  install_packages: true
{% if __grains__.get('osarch')|lower == "amd64" %}
  pin_version: true
  version: '3001*'
  py_ver: 'py3'
  release: '3001'
{% else %}
{# saltstack does not provide packages for !amd64, use debian packages #}
  lookup:
    pkgrepo: ''
{% endif %}

  minion:
    master:
      - __: overwrite
{% for master in masters %}
      - {{ master|json }}
{% endfor %}
    master_alive_interval: 30
    master_type: str
    retry_dns: 30

---

Bug details

Describe the bug

When wanting to update from 3001 to 3002, when modifying config as this :

diff --git a/pillars/global/salt.yml b/pillars/global/salt.yml
index 4ca398d..b84a321 100644
--- a/pillars/global/salt.yml
+++ b/pillars/global/salt.yml
@@ -8,9 +8,9 @@ salt:
   install_packages: true
 {% if __grains__.get('osarch')|lower == "amd64" %}
   pin_version: true
-  version: '3001*'
+  version: '3002*'
   py_ver: 'py3'
-  release: '3001'
+  release: '3002'
 {% else %}
 {# saltstack does not provide packages for !amd64, use debian packages #}
   lookup:

Minion wants to update itself from 3001 to 3002. The debian package is correctly updated. However, the debian/systemctl is not correctly reloaded.

A salt-minion process is running, but it's not the systemctl service.

Steps to reproduce the bug

Expected behaviour

Salt-minion should be able to update itself without crashing, or the formula should be able to handle this kind of update

Attempts to fix the bug

none

Additional context

/var/log/dpkg.log:

2020-11-05 12:35:47 startup archives unpack
2020-11-05 12:35:47 upgrade salt-minion:all 3001.3+ds-1 3002.1+ds-1
2020-11-05 12:35:47 status half-configured salt-minion:all 3001.3+ds-1
2020-11-05 12:35:47 status unpacked salt-minion:all 3001.3+ds-1
2020-11-05 12:35:47 status half-installed salt-minion:all 3001.3+ds-1
2020-11-05 12:35:47 status triggers-pending systemd:amd64 241-7~deb10u4
2020-11-05 12:35:47 status triggers-pending man-db:amd64 2.8.5-2
2020-11-05 12:35:47 status unpacked salt-minion:all 3002.1+ds-1
2020-11-05 12:35:47 upgrade salt-common:all 3001.3+ds-1 3002.1+ds-1
2020-11-05 12:35:47 status half-configured salt-common:all 3001.3+ds-1
2020-11-05 12:35:48 status unpacked salt-common:all 3001.3+ds-1
2020-11-05 12:35:48 status half-installed salt-common:all 3001.3+ds-1
2020-11-05 12:35:48 status unpacked salt-common:all 3002.1+ds-1
2020-11-05 12:35:49 startup packages configure
2020-11-05 12:35:49 configure salt-common:all 3002.1+ds-1 <none>
2020-11-05 12:35:49 status unpacked salt-common:all 3002.1+ds-1
2020-11-05 12:35:49 status half-configured salt-common:all 3002.1+ds-1
2020-11-05 12:35:52 status installed salt-common:all 3002.1+ds-1
2020-11-05 12:35:52 configure salt-minion:all 3002.1+ds-1 <none>
2020-11-05 12:35:52 status unpacked salt-minion:all 3002.1+ds-1
2020-11-05 12:35:52 status half-configured salt-minion:all 3002.1+ds-1
2020-11-05 12:37:23 status installed salt-minion:all 3002.1+ds-1
2020-11-05 12:37:23 trigproc man-db:amd64 2.8.5-2 <none>
2020-11-05 12:37:23 status half-configured man-db:amd64 2.8.5-2
2020-11-05 12:37:24 status installed man-db:amd64 2.8.5-2
2020-11-05 12:37:24 trigproc systemd:amd64 241-7~deb10u4 <none>
2020-11-05 12:37:24 status half-configured systemd:amd64 241-7~deb10u4
2020-11-05 12:37:24 status installed systemd:amd64 241-7~deb10u4

/var/log/salt/minion:

2020-11-05 12:37:27,641 [salt.state       :328 ][ERROR   ][9077] #### /etc/salt/minion.d/reactor.conf ####
Failed to cache salt://salt/files/minion.d/reactor.conf?saltenv=base: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/salt/minion.d/engine.conf ####
Failed to cache salt://salt/files/minion.d/engine.conf?saltenv=base: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/salt/minion.d/f_defaults.conf ####
Failed to cache salt://salt/files/minion.d/f_defaults.conf?saltenv=base: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/salt/minion.d/beacons.conf ####
Failed to cache salt://salt/files/minion.d/beacons.conf?saltenv=base: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:28,921 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,004 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/pre-install.d/10packagelist ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/pre-install.d/40uncommitted-changes-during-salt-run ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/pre-install.d/50uncommitted-changes ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,095 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/post-install.d/50vcs-commit ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/post-install.d/40vcs-commit-during-salt-run ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,198 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/init.d/20restore-etckeeper ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/70vcs-add ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/50vcs-ignore ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/40vcs-init ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/50vcs-perm ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/50vcs-pre-commit-hook ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/60darcs-deleted-symlinks ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/init.d/10restore-metadata ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,335 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/uninit.d/50vcs-uninit ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/uninit.d/01prompt ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/uninit.d/50remove-metadata ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,422 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/unclean.d/50test ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,499 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/update-ignore.d/01update-ignore ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,581 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/pre-commit.d/30store-metadata ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/pre-commit.d/20warn-problem-files ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,672 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/commit.d/30git-add ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/commit.d/30hg-addremove ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/commit.d/30bzr-add ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/commit.d/10vcs-test ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/commit.d/30darcs-add ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'

#### /etc/etckeeper/commit.d/50vcs-commit ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,768 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/vcs.d/50vcs-cmd ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,837 [salt.state       :328 ][ERROR   ][9077] #### /etc/etckeeper/list-installed.d/50list-installed ####
Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:37:29,936 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1741, in installed
    ) = result
ValueError: not enough values to unpack (expected 7, got 4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1751, in installed
    hold_ret = __salt__[action](name=name, pkgs=pkgs, sources=sources)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 1268, in unhold
    state = get_selections(pattern=target)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 2625, in get_selections
    stdout = __salt__["cmd.run_stdout"](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run_stdout'

2020-11-05 12:38:35,015 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1741, in installed
    ) = result
ValueError: not enough values to unpack (expected 7, got 4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1751, in installed
    hold_ret = __salt__[action](name=name, pkgs=pkgs, sources=sources)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 1268, in unhold
    state = get_selections(pattern=target)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 2625, in get_selections
    stdout = __salt__["cmd.run_stdout"](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run_stdout'

2020-11-05 12:39:41,098 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1741, in installed
    ) = result
ValueError: not enough values to unpack (expected 7, got 4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/pkg.py", line 1751, in installed
    hold_ret = __salt__[action](name=name, pkgs=pkgs, sources=sources)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 1268, in unhold
    state = get_selections(pattern=target)
  File "/usr/lib/python3/dist-packages/salt/modules/aptpkg.py", line 2625, in get_selections
    stdout = __salt__["cmd.run_stdout"](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run_stdout'

2020-11-05 12:39:43,307 [salt.state       :328 ][ERROR   ][9077] Unable to check_cmd file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:39:43,346 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,369 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,396 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,414 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,439 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,463 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,485 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,509 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/locale.py", line 79, in present
    if __salt__["locale.avail"](name):
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 277, in avail
    avail_locales = __salt__["locale.list_avail"]()
  File "/usr/lib/python3/dist-packages/salt/modules/localemod.py", line 139, in list_avail
    return __salt__["cmd.run"]("locale -a").split("\n")
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run'

2020-11-05 12:39:43,528 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:39:43,542 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:39:43,557 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:39:43,640 [salt.loader      :2000][ERROR   ][9077] Exception raised when processing __virtual__ function for salt.loaded.int.states.aws_sqs. Module will not be loaded: 'cmd.has_exec'
2020-11-05 12:39:43,646 [salt.loader      :2018][WARNING ][9077] salt.loaded.int.states.aws_sqs.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'aws_sqs', please fix this.
2020-11-05 12:39:43,849 [salt.loader      :2000][ERROR   ][9077] Exception raised when processing __virtual__ function for salt.loaded.int.states.hg. Module will not be loaded: 'cmd.has_exec'
2020-11-05 12:39:43,854 [salt.loader      :2018][WARNING ][9077] salt.loaded.int.states.hg.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'hg', please fix this.
2020-11-05 12:39:44,122 [salt.loader      :2000][ERROR   ][9077] Exception raised when processing __virtual__ function for salt.loaded.int.states.rabbitmq_plugin. Module will not be loaded: 'cmd.has_exec'
2020-11-05 12:39:44,128 [salt.loader      :2018][WARNING ][9077] salt.loaded.int.states.rabbitmq_plugin.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'rabbitmq_plugin', please fix this.
2020-11-05 12:39:44,230 [salt.loader      :2000][ERROR   ][9077] Exception raised when processing __virtual__ function for salt.loaded.int.states.svn. Module will not be loaded: 'cmd.has_exec'
2020-11-05 12:39:44,231 [salt.loader      :2018][WARNING ][9077] salt.loaded.int.states.svn.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'svn', please fix this.
2020-11-05 12:39:44,259 [salt.loader      :2000][ERROR   ][9077] Exception raised when processing __virtual__ function for salt.loaded.int.states.sysrc. Module will not be loaded: 'cmd.has_exec'
2020-11-05 12:39:44,259 [salt.loader      :2018][WARNING ][9077] salt.loaded.int.states.sysrc.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'sysrc', please fix this.
2020-11-05 12:39:44,379 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2118, in call
    ret.update(self._run_check(low))
  File "/usr/lib/python3/dist-packages/salt/state.py", line 897, in _run_check
    ret["result"] = _ret["result"]
  File "/usr/lib/python3/dist-packages/salt/state.py", line 1012, in _run_check_unless
    "result": True,
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.retcode'

2020-11-05 12:39:48,203 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
[ this message repeats several times]
2020-11-05 12:40:20,145 [salt.state       :328 ][ERROR   ][9077] Failed to cache salt://logrotate/templates/logrotate.conf.tmpl: cache_file() got an unexpected keyword argument 'verify_ssl'
[ this type of message repeats several times for other files ]
2020-11-05 12:40:20,878 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2118, in call
    ret.update(self._run_check(low))
  File "/usr/lib/python3/dist-packages/salt/state.py", line 890, in _run_check
    if "shell" in low_data:
  File "/usr/lib/python3/dist-packages/salt/state.py", line 949, in _run_check_onlyif
    elif cmd == 0:
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.retcode'

2020-11-05 12:40:22,993 [salt.state       :328 ][ERROR   ][9077] Unable to manage file: cache_file() got an unexpected keyword argument 'verify_ssl'
2020-11-05 12:40:23,027 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2118, in call
    ret.update(self._run_check(low))
  File "/usr/lib/python3/dist-packages/salt/state.py", line 890, in _run_check
    if "shell" in low_data:
  File "/usr/lib/python3/dist-packages/salt/state.py", line 949, in _run_check_onlyif
    elif cmd == 0:
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.retcode'

2020-11-05 12:40:23,086 [salt.state       :328 ][ERROR   ][9077] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/state.py", line 2154, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 2087, in wrapper
    virtual = virtual[0]
  File "/usr/lib/python3/dist-packages/salt/states/service.py", line 444, in running
    if not _available(name, ret):
  File "/usr/lib/python3/dist-packages/salt/states/service.py", line 354, in _available
    avail = __salt__["service.available"](name)
  File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 629, in available
    _check_for_unit_changes(name)
  File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 142, in _check_for_unit_changes
    if _untracked_custom_unit_found(name) or _unit_file_changed(name):
  File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 404, in _unit_file_changed
    status = _systemctl_status(name)["stdout"].lower()
  File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 366, in _systemctl_status
    __context__[contextkey] = __salt__["cmd.run_all"](
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
    #            self.clean_modules as that WILL prevent this loader object from being garbage
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'cmd.run_all'

systemctl status salt-minion:

# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/lib/systemd/system/salt-minion.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2020-11-05 12:37:28 CET; 1h 5min ago
     Docs: man:salt-minion(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
  Process: 11814 ExecStart=/usr/bin/salt-minion (code=exited, status=1/FAILURE)
 Main PID: 11814 (code=exited, status=1/FAILURE)
    Tasks: 8 (limit: 1162)
   Memory: 194.9M
   CGroup: /system.slice/salt-minion.service
           ├─6995 /usr/bin/python3 /usr/bin/salt-minion
           └─9077 /usr/bin/python3 /usr/bin/salt-minion

Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:     if _untracked_custom_unit_found(name) or _unit_file_changed(name):
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:   File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 404, in _unit_file_changed
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:     status = _systemctl_status(name)["stdout"].lower()
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:   File "/usr/lib/python3/dist-packages/salt/modules/systemd_service.py", line 366, in _systemctl_status
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:     __context__[contextkey] = __salt__["cmd.run_all"](
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:   File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:     #            self.clean_modules as that WILL prevent this loader object from being garbage
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:   File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]:     raise KeyError(key)
Nov 05 12:40:23 ip-172-31-11-45.eu-west-1.compute.internal salt-minion[6985]: KeyError: 'cmd.run_all'

a process is still running:

aws/preprod:root@ip-172-31-11-45: ~ # ps afx | grep salt
 6995 ?        S      0:04 /usr/bin/python3 /usr/bin/salt-minion
 9077 ?        Sl     0:42  \_ /usr/bin/python3 /usr/bin/salt-minion

but it's not responding :

root@salt-master# salt 'ip-172-31-11-45.eu-west-1.compute.internal' test.ping
ip-172-31-11-45.eu-west-1.compute.internal:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20201105124413784303
ERROR: Minions returned with non-zero exit code
root@salt-master# salt-run jobs.lookup_jid 20201105124413784303
root@salt-master#

[myii]

Probably useful for finding a resolution:

• https://docs.saltstack.com/en/latest/faq.html#what-is-the-best-way-to-restart-a-salt-minion-daemon-using-salt-after-upgrade
  • cmd.run_bg perhaps?

@javierbertoli Any thoughts?

[Yoda-BZH]

Probably useful for finding a resolution:

• https://docs.saltstack.com/en/latest/faq.html#what-is-the-best-way-to-restart-a-salt-minion-daemon-using-salt-after-upgrade

  • cmd.run_bg perhaps?

Shouldn't the salt-formula use this then ?