Open bsoldhogg opened 3 years ago
p3lim
commented
2 years ago Seeing the same with 3003.3 (latest on Fedora) and a HTTP(S) proxy, although with a master. I'm not seeing an endless loop however, it just spams this message with thousands of lines in the logs causing rotations to occur multiple time per highstate.
There is no functional issue for us, but we've had to place the salt-minion's logs in a blocklist on our log aggregator due to all the noise wasting our license quotas.
eliasp
commented
2 years ago Seeing it on 3005 when downloading binary files via a proxy.
How to reproduce (there might be a more straightforward way, I'm just mostly replicating what I'm running here for $reasons):
ssh -D 1080 localhost 'while true; do echo "$(date) Keep the proxy alive..."; sleep 5; done's2h: go install github.com/mritd/s2h@masters2h:
$ s2h
INFO[2022-27-24 15:31:08] Starting Socks5 Proxy Convert Server...
INFO[2022-27-24 15:31:08] HTTP Listen Address: 0.0.0.0:8081
INFO[2022-27-24 15:31:08] Socks5 Server Address: 127.0.0.1:1080 proxy_host: localhost
proxy_port: 8081 salt-call --local cp.get_url https://docs.saltproject.io/en/pdf/Salt-3005.pdf /tmp/Salt-3005.pdfThe result is: thousands of times this error repeated:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 1: invalid continuation byteIt works fine for non-binary files, e.g.:
salt-call --local cp.get_url https://saltproject.io/wp-content/uploads/2020/12/main-logo.svg /tmp/main-logo.svg
local:
/tmp/main-logo.svgVersion Report:
Salt Version:
Salt: 3005
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: Not Installed
Python: 3.8.10 (default, Jun 22 2022, 20:18:18)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-88-generic
system: Linux
version: Ubuntu 20.04 focalAfter finding https://github.com/tornadoweb/tornado/issues/1608 which was fixed in d7d9c467cda38f4c9352172ba7411edc29a85196, I followed the same approach and hot-fixed it here by doing this:
diff --git a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
index 8652343..6ef3349 100644
--- a/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py.orig
+++ b/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py
@@ -494,7 +494,7 @@ class CurlAsyncHTTPClient(AsyncHTTPClient):
def _curl_debug(self, debug_type, debug_msg):
debug_types = ('I', '<', '>', '<', '>')
- debug_msg = native_str(debug_msg)
+ debug_msg = native_str(debug_msg.decode('latin1'))
if debug_type == 0:
curl_log.debug('%s', debug_msg.strip())
elif debug_type in (1, 2):I'll have to look into making sure this fix is addressed properly upstream - will follow up here...
eliasp
commented
2 years ago As Salt ships a bundled Tornado curl_httpclient.py, it might be sufficient until upstream addresses this issue (https://github.com/tornadoweb/tornado/issues/3183) to apply the diff from my previous comment...
Description I'm testing a masterless Windows minion in an environment that must use a proxy to get out to the internet. During a
pkg.installrun, the minion attempts to download the package but it runs in to an unhandled exception for UnicodeDecodeError which ultimately results in an endless loop. Only when I kill the salt-call operation do I see the connection in the proxy logs.This behaviour is not seen with a masterless minion in a different environment that doesn't require a proxy.
Setup
Salt minion config (C:\salt\conf\minion.d\masterless.conf)
Test state to achieve a download of an installer (C:\Temp\packages\win\repo-ng\datadog-pkg.sls)
Debug output
Steps to Reproduce the behavior This behaviour can be reproduced with any installer, no matter if it's hosted on S3 or not. I'm not able to test without using a proxy on EC2 but I have tested in a VMware environment that doesn't require a proxy, with the same minion config (minus proxy settings), states and packages copied over as is, and there is no issue.
Powershell is able to use wget to download installers from the internet using environment proxy settings.
Expected behavior I expect Salt to be able to retrieve packages from the internet with proxy settings configured in the minion, just as it does without.
Versions Report