[BUG] TCP transport broken on 3004.1 and relevant bugfixes

[lukasraska]

Description Salt Master on TCP transport is broken on newly released version (https://saltproject.io/security_announcements/salt-security-advisory-release/) that mitigates several CVEs, resulting in no jobs being published.

In master logs following stacktrace can be seen:

2022-03-28 20:15:02,912 [tornado.application:640 ][ERROR   ][25004] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f354fda3ea0>, <sal
t.ext.tornado.concurrent.Future object at 0x7f35462c6940>)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 294, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib64/python3.6/types.py", line 248, in wrapped
    coro = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/transport/tcp.py", line 1565, in publish_payload
    payload = salt.transport.frame.frame_msg(package["payload"])
KeyError: 'payload'

Setup TCP transport on master & minion, 3004.1 master.

Please be as specific as possible and give set-up details.

• [ ] on-prem machine
• [x] VM (Virtualbox, KVM, etc. please specify)
• [ ] VM running on a cloud service, please be explicit and add details
• [ ] container (Kubernetes, Docker, containerd, etc. please specify)
• [ ] or a combination, please be explicit
• [ ] jails if it is FreeBSD

Steps to Reproduce the behavior Run any job, doesn't get published and no results can be obtained.

Expected behavior Jobs can actually be published.

Screenshots If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

```yaml Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.9.1 cherrypy: Not Installed dateutil: 2.4.2 docker-py: Not Installed gitdb: 0.6.4 gitpython: 1.0.1 Jinja2: 2.11.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.14 pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Aug 13 2020, 07:46:32) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.0.0 smmap: 0.9.0 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: rhel 7.9 Maipo locale: UTF-8 machine: x86_64 release: 3.10.0-1160.53.1.el7.x86_64 system: Linux version: Red Hat Enterprise Linux Server 7.9 Maipo ```

Additional context https://github.com/saltstack/salt/blob/v3004.1/salt/transport/tcp.py#L1564 should be package = self.pack_publish(package) (package instead of payload) - can be applied as a workaround

[dwoz]

@lukasraska Can you provide us a version report from both master and minion?

[lukasraska]

@dwoz master version report is under the Versions Report section, minion version isn't relevant (the payload is never sent)... so even simple salt -L master.tld test.ping is affected (so same version report as from master)

[dwoz]

@lukasraska I've confirmed this is an issue and your suggestion seems to resolve it.

From 21166b20f01ab9b49d4c43c6a19aa21ecba1d72a Mon Sep 17 00:00:00 2001
From: "Daniel A. Wozniak" <dwozniak@saltstack.com>
Date: Mon, 28 Mar 2022 14:11:50 -0700
Subject: [PATCH] Tcp transport bugfix

---
 salt/transport/tcp.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/salt/transport/tcp.py b/salt/transport/tcp.py
index f00b3c40eb..2c4844a2a6 100644
--- a/salt/transport/tcp.py
+++ b/salt/transport/tcp.py
@@ -1562,7 +1562,7 @@ class PubServer(salt.ext.tornado.tcpserver.TCPServer):
     def publish_payload(self, package, _):
         log.debug("TCP PubServer sending payload: %s", package)
         payload = self.pack_publish(package)
-        payload = salt.transport.frame.frame_msg(package["payload"])
+        payload = salt.transport.frame.frame_msg(payload["payload"])

         to_remove = []
         if "topic_lst" in package:
-- 
2.30.2

[lukasraska]

@dwoz great, I initially thought the package -> payload would be a way t go, but further down the topic_lst works around package, rather than the payload variable... so when the topic_lst content is list, it's working as intended, but when it's string, it will fail (because that's processed in the pack_publish), so following patch is probably for the best

From 1399acfb8cb466f13fbd8f7177653f27746d6ed7 Mon Sep 17 00:00:00 2001
From: Lukas Raska <lukas@raska.me>
Date: Mon, 28 Mar 2022 21:38:44 +0200
Subject: [PATCH] Assign packaged payload to proper variable in TCP transport
 layer

---
 salt/transport/tcp.py | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/salt/transport/tcp.py b/salt/transport/tcp.py
index f00b3c40eb6af..48ece5f83d350 100644
--- a/salt/transport/tcp.py
+++ b/salt/transport/tcp.py
@@ -1561,7 +1561,7 @@ def handle_stream(self, stream, address):
     @salt.ext.tornado.gen.coroutine
     def publish_payload(self, package, _):
         log.debug("TCP PubServer sending payload: %s", package)
-        payload = self.pack_publish(package)
+        package = self.pack_publish(package)
         payload = salt.transport.frame.frame_msg(package["payload"])

         to_remove = []

[dwoz]

@dwoz great, I initially thought the package -> payload would be a way t go, but further down the topic_lst works around package, rather than the payload variable... so when the topic_lst content is list, it's working as intended, but when it's string, it will fail (because that's processed in the pack_publish), so following patch is probably for the best

From 1399acfb8cb466f13fbd8f7177653f27746d6ed7 Mon Sep 17 00:00:00 2001
From: Lukas Raska <lukas@raska.me>
Date: Mon, 28 Mar 2022 21:38:44 +0200
Subject: [PATCH] Assign packaged payload to proper variable in TCP transport
 layer

---
 salt/transport/tcp.py | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/salt/transport/tcp.py b/salt/transport/tcp.py
index f00b3c40eb6af..48ece5f83d350 100644
--- a/salt/transport/tcp.py
+++ b/salt/transport/tcp.py
@@ -1561,7 +1561,7 @@ def handle_stream(self, stream, address):
     @salt.ext.tornado.gen.coroutine
     def publish_payload(self, package, _):
         log.debug("TCP PubServer sending payload: %s", package)
-        payload = self.pack_publish(package)
+        package = self.pack_publish(package)
         payload = salt.transport.frame.frame_msg(package["payload"])

         to_remove = []

Yes, you are correct. This is how it is on the 3003.4 branch which does not have the bug. Oddly enough 3002.8 does have the bug.

[cheburakshu]

During salt-bootstrap of the latest minion version we are seeing this error, is this traceback related as it comes from the same variable?

2022-03-29 11:54:26,543 [salt.utils.process:244 ][INFO    ][4107] pidfile: /var/run/process_responsibility_salt-minion.pid not found
2022-03-29 11:54:26,703 [salt.cli.daemons :89  ][INFO    ][4107] Starting up the Salt Minion
2022-03-29 11:54:26,704 [salt.utils.event :1125][INFO    ][4107] Starting pull socket on /var/run/salt/minion/minion_event_051400d64e_pull.ipc
2022-03-29 11:54:27,171 [salt.minion      :1292][INFO    ][4107] Creating minion process manager
2022-03-29 11:54:27,286 [salt.crypt       :896 ][INFO    ][4107] Generating keys: /etc/salt/pki/minion
2022-03-29 11:54:27,411 [tornado.application:640 ][ERROR   ][4107] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f947be52a60>, <salt.ext.tornado.concurrent.Future object at 0x7f947a502ba8>)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 654, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 780, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 792, in handle_signin_response
    clear_signature = payload["sig"]
KeyError: 'sig'
2022-03-29 11:55:27,222 [salt.minion      :1095][ERROR   ][4107] Minion unable to successfully connect to a Salt Master.```

[vdfdev]

Same error here as @cheburakshu, I am getting this error on a masterless salt after upgrading to 3004.1 yesterday:

2022-03-29 18:16:52,974 [tornado.application:640 ][ERROR   ][29690] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fbd00269378>, <salt.ext.tornado.concurrent.Future object at 0x7fbd0040[110/1068]
Traceback (most recent call last):                      
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()         
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper               
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()                         
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)                     
  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run                                 
    yielded = self.gen.throw(*exc_info)                                                                            
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 654, in _authenticate                                                                                                                                                      
    creds = yield self.sign_in(channel=channel)                                                                                                                                                                                        
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run                                                                                                                                                     
    value = future.result()                                                                                                                                                                                                            
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result                                                                                                                                            
    raise_exc_info(self._exc_info)                                                                                                                                                                                                     
  File "<string>", line 4, in raise_exc_info                                                                                                                                                                                           
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run                    
    yielded = self.gen.send(value)                                                                                                                                                                                                     
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 780, in sign_in                                                                                                                                                            
    ret = self.handle_signin_response(sign_in_payload, payload)                                                    
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 792, in handle_signin_response            
    clear_signature = payload["sig"]                                                                               
KeyError: 'sig'

After this error, the salt-call --local state.highstate seems to hang even though all states executed correctly. This was not happening prior to yesterday's release.

The annoying thing is that on debian11, the bootstrap script on https://bootstrap.saltproject.io/ does not have any flag that allows me to roll back to a version before yesterday's release as the only option is 3004... And 3004 was modified yesterday per https://repo.saltproject.io/py3/debian/11/amd64/3004

EDIT: we found out that the state that is causing this is the below:

restart-salt-minion:
  cmd.run:
    - name: 'salt-call service.restart salt-minion'
    - bg: True
    - onchanges:
      - file: /etc/salt/minion
      - file: /etc/salt/minion.d
      - pkg: salt-minion

Which is the recommended way of restarting the minion on https://docs.saltproject.io/en/latest/faq.html#restart-using-states

[whytewolf]

can we see some options that you are using to have tha thappen @vdfdev and @cheburakshu I don't seem to be able to replicate it. also what version is the master on? has it been updated first?

[weneve2]

so is there anyone could help me to solve this problem ... the master OS is ubuntu18.04 , salt version is Salt: 2017.7.4 the minion OS is ubuntu 20.04 , salt version is Salt: 3004.1 but the other minion (ubuntu 20.04 , Salt: 3004 ) is working well the salt lastest version could not work anyhow ...... and degraded met the same problem ...

[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key [DEBUG ] Closing AsyncZeroMQReqChannel instance [ERROR ] Exception in callback functools.partial(<function wrap..null_wrapper at 0x7f5a381a5e50>, <salt.ext.tornado.concurrent.Future object at 0x7f5a381a6430>) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback ret = callback() File "/usr/lib/python3/dist-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper return fn(*args, *kwargs) File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result future.result() File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1064, in run yielded = self.gen.throw(exc_info) File "/usr/lib/python3/dist-packages/salt/crypt.py", line 648, in _authenticate creds = yield self.sign_in(channel=channel) File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1056, in run value = future.result() File "/usr/lib/python3/dist-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/lib/python3/dist-packages/salt/ext/tornado/gen.py", line 1070, in run yielded = self.gen.send(value) File "/usr/lib/python3/dist-packages/salt/crypt.py", line 773, in sign_in ret = self.handle_signin_response(sign_in_payload, payload) File "/usr/lib/python3/dist-packages/salt/crypt.py", line 785, in handle_signin_response clear_signature = payload["sig"] KeyError: 'sig' ^C[DEBUG ] Closing IPCMessageSubscriber instance [WARNING ] Minion received a SIGINT. Exiting. [INFO ] Shutting down the Salt Minion

[lukasraska]

@weneve2 this is different issue, you need to update your master - 2017.7.4 is really old. See https://docs.saltproject.io/en/latest/topics/releases/3004.1.html - 3004.1 minions are not able to communicate with masters older than 3004.1. You must upgrade your masters before upgrading minions.

[vdfdev]

@whytewolf In my case we are using the packer masterless provisioner: https://www.packer.io/plugins/provisioners/salt which uses the bootstrap script to install the minion and salt the VM. I tried to reproduce this in a docker container but was unable to, so I dont know exactly what configuration is causing this for us. Disabling this restart only while salting the VM worked, and this workaround is good enough for us because we dont really need this restart on initial VM creation as it will be converted to an image anyway...

[whytewolf]

@vdfdev are you sure that you are not connecting to a master? the payload["sig"] error you are posting is normally the minion trying to parse the payload that came from the master. and not finding the sig. which only happens if the minion is newer than the master. if the minion isn't connecting to a master there shouldn't even be a payload to parse.

[vdfdev]

@vdfdev are you sure that you are not connecting to a master? the payload["sig"] error you are posting is normally the minion trying to parse the payload that came from the master. and not finding the sig. which only happens if the minion is newer than the master. if the minion isn't connecting to a master there shouldn't even be a payload to parse.

Ah, that makes sense then. We use the masterless provisioner while generating the VM image to setup our configuration so when the machine uses that image it can connect to the master. By restarting the minion service, it probably was trying to connect to the master and erroring out. In this step of generating the image with packer, it isn't desirable for us to connect to the master, so the workaround is good. I haven't had any issues connecting to the master after the image is created.

Ps: but thinking about it, we haven't upgraded our VMs to use the latest image, so I will do so to check if any issues happens. Our master might be outdated in relation to our minions indeed.

[pushkarsawant]

I am having the same issue. I have upgraded the master to node to latest. but still 'sig' is not part of master response.

This is the response from master. {'load': {'ret': True}, 'enc': 'clear'}

My minions with older version continue to work fine but i am unable to register any new minions.

[whytewolf]

@pushkar-engagio what version is your master running as?

[pushkarsawant]

All packages are on 3004.1-1. I am seeing the error on the minion running same as master nodes as well as any new nodes i try to join. ` yum list salt-* Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile

• base: download.cf.centos.org
• epel: iad.mirror.rackspace.com
• extras: download.cf.centos.org
• updates: download.cf.centos.org Installed Packages salt.noarch 3004.1-1.el7 @salt-latest-repo salt-api.noarch 3004.1-1.el7 @salt-latest-repo salt-cloud.noarch 3004.1-1.el7 @salt-latest-repo salt-master.noarch 3004.1-1.el7 @salt-latest-repo salt-minion.noarch 3004.1-1.el7 @salt-latest-repo salt-ssh.noarch 3004.1-1.el7 @salt-latest-repo salt-syndic.noarch 3004.1-1.el7 @salt-latest-repo`

[pushkarsawant]

This is the error that i see in salt-minion logs

Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: [ERROR   ] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f0a967c5048>, <salt.ext.tornado.concurrent.Future object at 0x7f0a9c069160>)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: Traceback (most recent call last):
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: ret = callback()
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: return fn(*args, **kwargs)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: future.result()
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: raise_exc_info(self._exc_info)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "<string>", line 4, in raise_exc_info
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: yielded = self.gen.throw(*exc_info)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 654, in _authenticate
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: creds = yield self.sign_in(channel=channel)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: value = future.result()
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: raise_exc_info(self._exc_info)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "<string>", line 4, in raise_exc_info
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: yielded = self.gen.send(value)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 780, in sign_in
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: ret = self.handle_signin_response(sign_in_payload, payload)
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 792, in handle_signin_response
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: clear_signature = payload["sig"]
Apr 04 20:09:36 ip-172-31-0-104. salt-minion[3029]: KeyError: 'sig'
Apr 04 20:10:36 ip-172-31-0-104. salt-minion[3029]: [ERROR   ] Minion unable to successfully connect to a Salt Master.

[whytewolf]

shutdown your master and make sure there are no other master processes running. then restart your master.

[pushkarsawant]

all the master were running post upgrade. same issue after stopping master service and starting master.

[whytewolf]

just to check. what does line 775 of salt/transport/zeromq.py say on the master?

[pushkarsawant]

nm. found the problem. After the upgrade my minion configuration was erased. Updated the minion file and restarted. the minion is back online now.

[weneve2]

@weneve2 this is different issue, you need to update your master - 2017.7.4 is really old. See https://docs.saltproject.io/en/latest/topics/releases/3004.1.html - 3004.1 minions are not able to communicate with masters older than 3004.1. You must upgrade your masters before upgrading minions.

oh , i never check the release note , just initialize the server and install the lastest version of saltstack ...... maybe highlight the information on install guide page can reduce some confuse

[kymikoloco]

I am also having the original issue without upgrading the master.

This minion install is fresh using the https://github.com/saltstack/salt-bootstrap#install-on-windows Powershell instructions. Regardless of the user I connect as, whether as service or Administrator on using the salt-minion-debug.bat files, I get the same exception in crypt.py

Salt Version (master):
          Salt: 3003.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 3.0.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: 1.1.0
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: 1.4.4
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Jun  2 2021, 10:49:15)
  python-gnupg: 0.4.5
        PyYAML: 5.4.1
         PyZMQ: 22.2.1
         smmap: 2.0.5
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 4.15.0-153-generic
        system: Linux
       version: Ubuntu 20.04 focal

Salt Version (minion):
          Salt: 3003.4

Dependency Versions:
          cffi: 1.14.5
      cherrypy: 18.6.0
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
  python-gnupg: 0.4.6
        PyYAML: 5.4.1
         PyZMQ: 18.0.1
         smmap: 3.0.4
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.19041 SP0

2022-04-15 17:42:38,693 [tornado.application:640 ][ERROR   ][1424] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x000002582040D318>, <salt.ext.tornado.concurrent.Future object at 0x0000025820C0A108>)
Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\ioloop.py", line 606, in _run_callback
    ret = callback()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\ioloop.py", line 628, in _discard_future_result
    future.result()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 648, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 773, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 785, in handle_signin_response
    clear_signature = payload["sig"]
KeyError: 'sig'

[scotsie]

Just in case it helps with weight, prioritization, can confirm running into the OPs same error on salt-master 3004.1 and manual modification as noted in the pull request resolved the issue for me as well.

Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f786c74dd90>, <salt.ext.tornado.concurrent.Future object at 0x7f786c7a0e10>)
  Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 606, in _run_callback
    ret = callback()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 628, in _discard_future_result
    future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 294, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib64/python3.6/types.py", line 248, in wrapped
    coro = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/transport/tcp.py", line 1565, in publish_payload
    payload = salt.transport.frame.frame_msg(package["payload"])
 KeyError: 'payload'

[kymikoloco]

This looks like it was introduced in: https://github.com/saltstack/salt/commit/a1b0567a08aeafc41e8210b52c458c13e209343e

Installer for 3003.2 is no longer listed in https://repo.saltproject.io/windows/ or in https://repo.saltproject.io/windows/archive so I can't downgrade.

[whytewolf]

I am also having the original issue without upgrading the master.

This minion install is fresh using the https://github.com/saltstack/salt-bootstrap#install-on-windows Powershell instructions. Regardless of the user I connect as, whether as service or Administrator on using the salt-minion-debug.bat files, I get the same exception in crypt.py

Salt Version (master):
          Salt: 3003.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 3.0.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: 1.1.0
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: 1.4.4
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Jun  2 2021, 10:49:15)
  python-gnupg: 0.4.5
        PyYAML: 5.4.1
         PyZMQ: 22.2.1
         smmap: 2.0.5
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 4.15.0-153-generic
        system: Linux
       version: Ubuntu 20.04 focal

Salt Version (minion):
          Salt: 3003.4

Dependency Versions:
          cffi: 1.14.5
      cherrypy: 18.6.0
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
  python-gnupg: 0.4.6
        PyYAML: 5.4.1
         PyZMQ: 18.0.1
         smmap: 3.0.4
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 10
        system: Windows
       version: 10 10.0.19041 SP0

2022-04-15 17:42:38,693 [tornado.application:640 ][ERROR   ][1424] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x000002582040D318>, <salt.ext.tornado.concurrent.Future object at 0x0000025820C0A108>)
Traceback (most recent call last):
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\ioloop.py", line 606, in _run_callback
    ret = callback()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\stack_context.py", line 278, in null_wrapper
    return fn(*args, **kwargs)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\ioloop.py", line 628, in _discard_future_result
    future.result()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 648, in _authenticate
    creds = yield self.sign_in(channel=channel)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1056, in run
    value = future.result()
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\ext\tornado\gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 773, in sign_in
    ret = self.handle_signin_response(sign_in_payload, payload)
  File "c:\salt\bin\lib\site-packages\salt-3003.4-py3.7.egg\salt\crypt.py", line 785, in handle_signin_response
    clear_signature = payload["sig"]
KeyError: 'sig'

Your error is not the one in this issue. Your issue is you are trying to run a CVE-fixed minion against a Non CVE-fixed master.

Upgrade your master.

[PeterS242]

This looks like it was introduced in: a1b0567

Installer for 3003.2 is no longer listed in https://repo.saltproject.io/windows/ or in https://repo.saltproject.io/windows/archive so I can't downgrade.

@kymikoloco it is not well documented (the information is buried pretty well) but the real or correct location for archived installation files is here: https://archive.repo.saltproject.io/ . HTH

[marcraft2]

Same problem today. I'm in 3002.6 on the master and the minion though.