Open v0101 opened 1 year ago
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!
It does follow redirects, but you are not being given one.
Salt is receiving a 599, with a report that a proxy has returned a 400. There's no 302 in any of that.
Check the proxy settings that you're giving to Salt, as that's probably where the issue is.
Or possibly with how you've configured the pycurl
you built.
Sometimes GitHub also exceeds its Azure quota, causing release downloads to fail for a bit, but I forget which response code you get for that.
The 302 i get when i run curl -I https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
The proxy settings are the same in /etc/salt/minion.d/00-proxy.conf
and in the environment for curl. So I am a bit confused about the 599.
pycurl was built by pip, so I did not set any special configuration for it.
I also tried it with https://dl.k8s.io/release/v1.24.4/bin/linux/amd64/kubelet so I don't think azure is the issue here.
I did a tcpdump of both the curl command line and the salt run. In both i get the 302 redirect: curl
Frame 33: 1886 bytes on wire (15088 bits), 1886 bytes captured (15088 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 48966, Seq: 4115, Ack: 975, Len: 1826
Source Port: 8080
Destination Port: 48966
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (31)]
[TCP Segment Len: 1826]
Sequence Number: 4115 (relative sequence number)
Sequence Number (raw): 153878155
[Next Sequence Number: 5941 (relative sequence number)]
Acknowledgment Number: 975 (relative ack number)
Acknowledgment number (raw): 2486490739
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 62
[Calculated window size: 31744]
[Window size scaling factor: 512]
Checksum: 0x41c7 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (1826 bytes)
TCP segment data (368 bytes)
[PDU Size: 2784]
[2 Reassembled TCP Segments (1392 bytes): #31(1024), #33(368)]
[Frame: 31, payload: 0-1023 (1024 bytes)]
[Frame: 33, payload: 1024-1391 (368 bytes)]
[Segment count: 2]
[Reassembled TCP length: 1392]
[Reassembled TCP Data: 170303056b21223140a3af478f211e30dea19f500ed8db3d43d59769e2e62f60d7049087…]
Hypertext Transfer Protocol
[Proxy-Connect-Hostname: github.com]
[Proxy-Connect-Port: 443]
Transport Layer Security
TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
Opaque Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 1387
[Content Type: Application Data (23)]
Encrypted Application Data: 21223140a3af478f211e30dea19f500ed8db3d43d59769e2e62f60d7049087f514d212f4…
[Application Data Protocol: HyperText Transfer Protocol 2]
TLS segment data (1370 bytes)
Hypertext Transfer Protocol
[Proxy-Connect-Hostname: github.com]
[Proxy-Connect-Port: 443]
Transport Layer Security
TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
Opaque Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 1387
[Content Type: Application Data (23)]
Encrypted Application Data: cdfc17be98adb86e46f957ca824435ea3ce0b0f8c21e791730d9d4ae1c9b015811f5b169…
[Application Data Protocol: HyperText Transfer Protocol 2]
TLS segment data (1370 bytes)
TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
Opaque Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 61
[Content Type: Application Data (23)]
Encrypted Application Data: 41ba60060b6c1b5a4bc994feae10a4fc28a73093c04d20e16dad7798d68ddef92462594c…
[Application Data Protocol: HyperText Transfer Protocol 2]
TLS segment data (44 bytes)
[3 Reassembled TLS segments (2784 bytes): #33(1370), #33(1370), #33(44)]
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 1, Length 2775, 302 Found
Length: 2775
Type: HEADERS (1)
Flags: 0x05, End Headers, End Stream
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
[Pad Length: 0]
Header Block Fragment: 4803333032760a4769744875622e636f6d611d5475652c20303620446563203230323220…
[Header Length: 2948]
[Header Count: 15]
Header: :status: 302 Found
Header: server: GitHub.com
Header: date: Tue, 06 Dec 2022 15:06:50 GMT
Header: content-type: text/html; charset=utf-8
Header: vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
Header: location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/80172100/6a7ce7c5-7c3f-4b2b-b0b0-fe5fb498a36c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221206%2Fus-east-1%2Fs3
Header: cache-control: no-cache
Header: strict-transport-security: max-age=31536000; includeSubdomains; preload
Header: x-frame-options: deny
Header: x-content-type-options: nosniff
Header: x-xss-protection: 0
Header: referrer-policy: no-referrer-when-downgrade
Header: content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubuserconte
Header: content-length: 0
Header: x-github-request-id: F275:11E8:D86B98D:DF88AE1:638F5A89
salt
Frame 32: 426 bytes on wire (3408 bits), 426 bytes captured (3408 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 55210, Seq: 5554, Ack: 1012, Len: 366
Source Port: 8080
Destination Port: 55210
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (31)]
[TCP Segment Len: 366]
Sequence Number: 5554 (relative sequence number)
Sequence Number (raw): 3311262161
[Next Sequence Number: 5920 (relative sequence number)]
Acknowledgment Number: 1012 (relative ack number)
Acknowledgment number (raw): 1770980370
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 64
[Calculated window size: 32768]
[Window size scaling factor: 512]
Checksum: 0xd8f0 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (366 bytes)
TCP segment data (300 bytes)
[PDU Size: 2784]
[2 Reassembled TCP Segments (1392 bytes): #30(1092), #32(300)]
[Frame: 30, payload: 0-1091 (1092 bytes)]
[Frame: 32, payload: 1092-1391 (300 bytes)]
[Segment count: 2]
[Reassembled TCP length: 1392]
[Reassembled TCP Data: 170303056beb18f14112b837aad576cd505e1fdfef5a1e55d9dbcd994c0ce15cd6d4d83c…]
Hypertext Transfer Protocol
[Proxy-Connect-Hostname: github.com]
[Proxy-Connect-Port: 443]
Transport Layer Security
TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
Opaque Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 1387
[Content Type: Application Data (23)]
Encrypted Application Data: eb18f14112b837aad576cd505e1fdfef5a1e55d9dbcd994c0ce15cd6d4d83c8ab1360acf…
[Application Data Protocol: HyperText Transfer Protocol 2]
TLS segment data (1370 bytes)
Hypertext Transfer Protocol
[Proxy-Connect-Hostname: github.com]
[Proxy-Connect-Port: 443]
Transport Layer Security
TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
Opaque Type: Application Data (23)
Version: TLS 1.2 (0x0303)
Length: 61
[Content Type: Application Data (23)]
Encrypted Application Data: c779e280904524172801e5a7146e4cfd9bb551d1f48ecee463df6948fe5d7f744d859b48…
[Application Data Protocol: HyperText Transfer Protocol 2]
TLS segment data (44 bytes)
[3 Reassembled TLS segments (2784 bytes): #30(1370), #32(1370), #32(44)]
HyperText Transfer Protocol 2
Stream: HEADERS, Stream ID: 1, Length 2775, 302 Found
Length: 2775
Type: HEADERS (1)
Flags: 0x05, End Headers, End Stream
0... .... .... .... .... .... .... .... = Reserved: 0x0
.000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
[Pad Length: 0]
Header Block Fragment: 4803333032760a4769744875622e636f6d611d5475652c20303620446563203230323220…
[Header Length: 2948]
[Header Count: 15]
Header: :status: 302 Found
Header: server: GitHub.com
Header: date: Tue, 06 Dec 2022 14:46:58 GMT
Header: content-type: text/html; charset=utf-8
Header: vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
Header: location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/80172100/6a7ce7c5-7c3f-4b2b-b0b0-fe5fb498a36c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221206%2Fus-east-1%2Fs3
Header: cache-control: no-cache
Header: strict-transport-security: max-age=31536000; includeSubdomains; preload
Header: x-frame-options: deny
Header: x-content-type-options: nosniff
Header: x-xss-protection: 0
Header: referrer-policy: no-referrer-when-downgrade
Header: content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubuserconte
Header: content-length: 0
Header: x-github-request-id: E9FF:4EDE:D097418:D77CB4A:638F561B
But the next request fails with salt. Redirected curl
Frame 38: 210 bytes on wire (1680 bits), 210 bytes captured (1680 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.241.232, Dst: 172.30.240.100
Transmission Control Protocol, Src Port: 48968, Dst Port: 8080, Seq: 1, Ack: 1, Len: 150
Source Port: 48968
Destination Port: 8080
[Stream index: 2]
[Conversation completeness: Complete, WITH_DATA (31)]
[TCP Segment Len: 150]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 2498721712
[Next Sequence Number: 151 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 814931773
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 502
[Calculated window size: 64256]
[Window size scaling factor: 128]
Checksum: 0x3b3b [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (150 bytes)
Hypertext Transfer Protocol
CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n
[Expert Info (Chat/Sequence): CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
[CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: CONNECT
Request URI: objects.githubusercontent.com:443
Request Version: HTTP/1.1
Host: objects.githubusercontent.com:443\r\n
User-Agent: curl/7.76.1\r\n
Proxy-Connection: Keep-Alive\r\n
\r\n
[Full request URI: objects.githubusercontent.com:443]
[HTTP request 1/1]
[Response in frame: 40]
Redirected salt
Frame 37: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.241.232, Dst: 172.30.240.100
Transmission Control Protocol, Src Port: 55214, Dst Port: 8080, Seq: 1, Ack: 1, Len: 130
Source Port: 55214
Destination Port: 8080
[Stream index: 2]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 130]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 3937199183
[Next Sequence Number: 131 (relative sequence number)]
Acknowledgment Number: 1 (relative ack number)
Acknowledgment number (raw): 2294597533
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
Window: 502
[Calculated window size: 64256]
[Window size scaling factor: 128]
Checksum: 0x3b27 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (130 bytes)
Hypertext Transfer Protocol
CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n
[Expert Info (Chat/Sequence): CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
[CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
[Severity level: Chat]
[Group: Sequence]
Request Method: CONNECT
Request URI: objects.githubusercontent.com:443
Request Version: HTTP/1.1
Host: objects.githubusercontent.com:443\r\n
(nil)Proxy-Connection: Keep-Alive\r\n
[Expert Info (Warning/Protocol): Illegal characters found in header name]
[Illegal characters found in header name]
[Severity level: Warning]
[Group: Protocol]
\r\n
[Full request URI: objects.githubusercontent.com:443]
[HTTP request 1/1]
[Response in frame: 38]
It looks like something is wrong with the Proxy-Connection
header is wrong.
The response is then the 400
Frame 38: 247 bytes on wire (1976 bits), 247 bytes captured (1976 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 55214, Seq: 1, Ack: 131, Len: 187
Source Port: 8080
Destination Port: 55214
[Stream index: 2]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 187]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 2294597533
[Next Sequence Number: 189 (relative sequence number)]
Acknowledgment Number: 131 (relative ack number)
Acknowledgment number (raw): 3937199313
0101 .... = Header Length: 20 bytes (5)
Flags: 0x011 (FIN, ACK)
Window: 60
[Calculated window size: 30720]
[Window size scaling factor: 512]
Checksum: 0x42d6 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (187 bytes)
Hypertext Transfer Protocol
HTTP/1.0 400 Bad request\r\n
[Expert Info (Chat/Sequence): HTTP/1.0 400 Bad request\r\n]
[HTTP/1.0 400 Bad request\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.0
Status Code: 400
[Status Code Description: Bad Request]
Response Phrase: Bad request
Cache-Control: no-cache\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
[HTTP response 1/1]
[Time since request: 0.000507000 seconds]
[Request in frame: 37]
[Request URI: objects.githubusercontent.com:443]
File Data: 90 bytes
Line-based text data: text/html (3 lines)
In the salt-minion log file i see the following errors:
2022-12-06 16:11:32,025 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3: invalid start byte
2022-12-06 16:11:32,054 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 6: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 6: invalid start byte
2022-12-06 16:11:32,055 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 3: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 3: invalid start byte
2022-12-06 16:11:32,057 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 13: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 13: invalid start byte
2022-12-06 16:11:32,058 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 9: invalid continuation byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 9: invalid continuation byte
2022-12-06 16:11:32,059 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 5: invalid continuation byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 5: invalid continuation byte
2022-12-06 16:11:32,065 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
2022-12-06 16:11:32,066 [salt._logging.impl:1069][ERROR ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
Traceback (most recent call last):
File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
debug_msg = native_str(debug_msg)
File "salt/ext/tornado/escape.py", line 219, in to_unicode
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
Ok, so it did follow the redirect, so that's not the problem.
It appears something has inserted a NUL byte into the headers of Salt's request to your proxy, which would correctly result in a 400 error.
Since salt-pip install pycurl
would install the same version, I don't think that fixing #62949 will fix this issue.
I updated the title.
The same version of pycurl, but not necessarily built with the same version or configuration of libcurl and its other dependencies.
Ok, I see. So I guess I have to wait for #62949 to be fixed?
Or details of what your proxy is and how it's configured so it might be possible to reproduce.
The version we are using is squid 4.12. What details about the configuration would you need?
We also experience this, any news on possible fix?
Description As a workaround for #62949 I created the wheel file for pycurl and installed it via salt-pip. But when I try to download a file, e.g. https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz i get
Error: HTTP 599: Received HTTP code 400 from proxy after CONNECT reading /kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
This link accessed via curl gives a 302 redirect. For some reason, the redirect is not being followed. On previous version 3004.2 (non-onedir) this worked.Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
Steps to Reproduce the behavior I use Almalinux 9, master is 3004.2 and minion is 3005.1
In the state file i then try to download the file:
Expected behavior The file is downloaded.
Versions Report