search

pivotal-cf / pivnet-cli

CLI to interact with Tanzu Network API V2 interface.
Apache License 2.0
34 stars 27 forks source link

"during GET unexpected status code was returned: 200" when using Proxy #42

Open sluetze opened 4 years ago

sluetze commented 4 years ago

Hi all,

I'm using pivnet behind a proxy with SSL-Interception. This works for me, since i have added the proxy certificates to the ca-trust of the operating system.

The proxy is using authentication thus i defined 

export http_proxy=http://$username:$password@$proxy
export https_proxy=http://$username:$password@$proxy

I can use pivnet for alle actions except downloading products. I get the following error

 0 B / 940.50 MiB [-------------------------------------------------]   0.00% 7s
Downloader.Get: problem while waiting for chunks to download: failed during retryable request: during GET unexpected status code was returned: 200

Using wget on the download url I get an proxy request message as last feedback, before the download successfully starts:

proxy request send, waiting for answer ... 200 OK

I suspect this proxy answer to be an unhandled response in pivnet download-product-files

I verified the issue with pivnet in the versions

I verified that the issue does not exist, when not using a proxy

cjnosal commented 4 years ago

Does your proxy have caching enabled? Does the 200 response include the full 940.50 MiB?

The downloader makes several requests for partial ranges, so it expects the responses to be HTTP 206 (Partial Content) https://github.com/pivotal-cf/go-pivnet/blob/master/download/downloader.go#L189

A caching proxy may combine ranges and return a full response (https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.8) so we may need to ensure Downloader handles that case.

sluetze commented 4 years ago

Does your proxy have caching enabled?

yes, kind of. I am not able to access the proxy and configs (other team), but the proxy works roughly as follows:

  1. it establishes the connection
  2. it downloads the file locally to the proxy. while doing this the client only gets small byte-chunks so the connection is not lost
  3. after finishing the download locally, AV-Scans/security checks are performed
  4. after successfully checking the file, the proxy sends the full file to the client

Does the 200 response include the full 940.50 MiB?

I can guarantee, that this is NOT the case.

following a larger log from wget in debug mode (ips, hostnames, company-data redacted) where you can see the last two 200 codes. I suspect one of them to be unhandled. (I'm sorry for the german in the log, it's a system setting :-( )

The 206 (partial content) should be later (while the real download is ongoing).

I may be able to do a packet-capture while downloading with pivnet if that helps.

wget -d -O "harbor_v${harbor_version}.pivotal2" --header "Authorization: Bearer $access_token" ${harbor_download_link}
Setting --output-document (outputdocument) to harbor_v2.0.3.pivotal2
Setting --header (header) to Authorization: Bearer "<REDACTED>"
DEBUG output created by Wget 1.14 on linux-gnu.

URI encoding = »UTF-8«
URI encoding = »UTF-8«
Converted file name 'download' (UTF-8) -> 'download' (UTF-8)
--2020-10-09 08:58:50--  https://network.pivotal.io/api/v2/products/harbor-container-registry/releases/749813/product_files/797038/download
Auflösen des Hostnamen »<REDACTED>
Caching <REDACTED FQDN> => <REDACTED IPs>
Verbindungsaufbau zu <REDACTED>:80... verbunden.
Created socket 4.
Releasing 0x0000000000f431d0 (new refcount 1).

---request begin---
CONNECT network.pivotal.io:443 HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Proxy-Authorization: Basic <REDACTED>
Host: network.pivotal.io:443

---request end---
proxy responded with: [HTTP/1.0 200 Connection established

]
Initiating SSL handshake.
Handshake successful; connected socket 4 to SSL handle 0x0000000000f60cf0
certificate:
  subject: /CN=network.pivotal.io
  issuer:  <REDACTED>
X509 certificate successfully verified and matches host network.pivotal.io

---request begin---
GET /api/v2/products/harbor-container-registry/releases/749813/product_files/797038/download HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: network.pivotal.io
Connection: Close
Proxy-Connection: Keep-Alive
Authorization: Bearer "<REDACTED>

---request end---
Proxy-Anforderung gesendet, warte auf Antwort...
---response begin---
HTTP/1.1 302 Foun
Date: Fri, 09 Oct 2020 06:57:53 GMT
Vary: Accept-Encoding
Location: https://d13k9s5899twdr.cloudfront.net/partner-product-files/vmware-harbor/034a6270-64ca-4ab7-9cd4-924510d8f2aa?Expires=1602313073&Signature=a9UCikVkPD830x%7Eiu3dwVvvOUqtmqgG-dlV7ZvgqrMd2FeOjjhWOHU0CVSph4RpKCsn0xId47libvyquo1ykNAY%7EybFCZ%7ES-z0FuPPsjhJBYnuM9P7DZLpiCLbKOXlY3E6D3SIvdCQ-%7EoHu38mFDgCJIG5wuxTWxcBc1pZqESzIYIO3XZSNQb-sEDtseKvYtMW0KdJR-J-K-TKVO86uI2ScM-RjsLP4wa8JhJivf2u0QZFQCnybewWcEJRQrrdAKI4AIpZbVzgP66vkjlFo4oasy5DzaBlVnwX2dPlil9VxgJggXGoMQ3KRDc0-bWlmzMg78LIakVA6AMRKN5ViVew__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=034a6270-64ca-4ab7-9cd4-924510d8f2aa
X-Runtime: 0.103091
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
X-Request-Id: 7b6cd395-7dce-414f-8dc5-0acad6fa5af3
Cache-Control: no-cache
Referrer-Policy: strict-origin-when-cross-origin
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Transfer-Encoding: chunked
X-Vcap-Request-Id: c1889f9b-f27b-4c70-4724-d21b11f968ed
X-Download-Options: noopen
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Permitted-Cross-Domain-Policies: none

---response end---
302 Foun
Registered socket 4 for persistent reuse.
URI content encoding = »utf-8«
Platz: https://d13k9s5899twdr.cloudfront.net/partner-product-files/vmware-harbor/034a6270-64ca-4ab7-9cd4-924510d8f2aa?Expires=1602313073&Signature=a9UCikVkPD830x%7Eiu3dwVvvOUqtmqgG-dlV7ZvgqrMd2FeOjjhWOHU0CVSph4RpKCsn0xId47libvyquo1ykNAY%7EybFCZ%7ES-z0FuPPsjhJBYnuM9P7DZLpiCLbKOXlY3E6D3SIvdCQ-%7EoHu38mFDgCJIG5wuxTWxcBc1pZqESzIYIO3XZSNQb-sEDtseKvYtMW0KdJR-J-K-TKVO86uI2ScM-RjsLP4wa8JhJivf2u0QZFQCnybewWcEJRQrrdAKI4AIpZbVzgP66vkjlFo4oasy5DzaBlVnwX2dPlil9VxgJggXGoMQ3KRDc0-bWlmzMg78LIakVA6AMRKN5ViVew__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=034a6270-64ca-4ab7-9cd4-924510d8f2aa[folge]
Skipping 512 bytes of body: [<html><body>You are being <a href="https://d13k9s5899twdr.cloudfront.net/partner-product-files/vmware-harbor/034a6270-64ca-4ab7-9cd4-924510d8f2aa?Expires=1602313073&amp;Signature=a9UCikVkPD830x%7Eiu3dwVvvOUqtmqgG-dlV7ZvgqrMd2FeOjjhWOHU0CVSph4RpKCsn0xId47libvyquo1ykNAY%7EybFCZ%7ES-z0FuPPsjhJBYnuM9P7DZLpiCLbKOXlY3E6D3SIvdCQ-%7EoHu38mFDgCJIG5wuxTWxcBc1pZqESzIYIO3XZSNQb-sEDtseKvYtMW0KdJR-J-K-TKVO86uI2ScM-RjsLP4wa8JhJivf2u0QZFQCnybewWcEJRQrrdAKI4AIpZbVzgP66vkjlFo4oasy5DzaBlVnwX2dPlil9VxgJggXGoMQ3KRDc0-bWlmzMg78LSkipping 137 bytes of body: [IakVA6AMRKN5ViVew__&amp;Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&amp;filename=034a6270-64ca-4ab7-9cd4-924510d8f2aa">redirected</a>.</body></html>] done.
URI content encoding = None
URI encoding = »UTF-8«
Converted file name 'download' (UTF-8) -> 'download' (UTF-8)
--2020-10-09 08:58:50--  https://d13k9s5899twdr.cloudfront.net/partner-product-files/vmware-harbor/034a6270-64ca-4ab7-9cd4-924510d8f2aa?Expires=1602313073&Signature=a9UCikVkPD830x%7Eiu3dwVvvOUqtmqgG-dlV7ZvgqrMd2FeOjjhWOHU0CVSph4RpKCsn0xId47libvyquo1ykNAY%7EybFCZ%7ES-z0FuPPsjhJBYnuM9P7DZLpiCLbKOXlY3E6D3SIvdCQ-%7EoHu38mFDgCJIG5wuxTWxcBc1pZqESzIYIO3XZSNQb-sEDtseKvYtMW0KdJR-J-K-TKVO86uI2ScM-RjsLP4wa8JhJivf2u0QZFQCnybewWcEJRQrrdAKI4AIpZbVzgP66vkjlFo4oasy5DzaBlVnwX2dPlil9VxgJggXGoMQ3KRDc0-bWlmzMg78LIakVA6AMRKN5ViVew__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=034a6270-64ca-4ab7-9cd4-924510d8f2aa
Found <REDACTED> in host_name_addresses_map (0xf431d0)
Verbindungsaufbau zu <REDACTED>:80... verbunden.
Created socket 5.
Releasing 0x0000000000f431d0 (new refcount 1).

---request begin---
CONNECT d13k9s5899twdr.cloudfront.net:443 HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Proxy-Authorization: Basic <REDACTED>
Host: d13k9s5899twdr.cloudfront.net:443

---request end---
proxy responded with: [HTTP/1.0 200 Connection established

]
Initiating SSL handshake.
Handshake successful; connected socket 5 to SSL handle 0x0000000000f737b0
certificate:
  subject: /C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=d13k9s5899twdr.cloudfront.net
  issuer:  <REDACTED>
X509 certificate successfully verified and matches host d13k9s5899twdr.cloudfront.net

---request begin---
GET /partner-product-files/vmware-harbor/034a6270-64ca-4ab7-9cd4-924510d8f2aa?Expires=1602313073&Signature=a9UCikVkPD830x%7Eiu3dwVvvOUqtmqgG-dlV7ZvgqrMd2FeOjjhWOHU0CVSph4RpKCsn0xId47libvyquo1ykNAY%7EybFCZ%7ES-z0FuPPsjhJBYnuM9P7DZLpiCLbKOXlY3E6D3SIvdCQ-%7EoHu38mFDgCJIG5wuxTWxcBc1pZqESzIYIO3XZSNQb-sEDtseKvYtMW0KdJR-J-K-TKVO86uI2ScM-RjsLP4wa8JhJivf2u0QZFQCnybewWcEJRQrrdAKI4AIpZbVzgP66vkjlFo4oasy5DzaBlVnwX2dPlil9VxgJggXGoMQ3KRDc0-bWlmzMg78LIakVA6AMRKN5ViVew__&Key-Pair-Id=APKAIOIREZ2UWYGOZQDA&filename=034a6270-64ca-4ab7-9cd4-924510d8f2aa HTTP/1.1
User-Agent: Wget/1.14 (linux-gnu)
Accept: */*
Host: d13k9s5899twdr.cloudfront.net
Connection: Close
Proxy-Connection: Keep-Alive
Authorization: Bearer "<REDACTED>"
---request end---
Proxy-Anforderung gesendet, warte auf Antwort...
---response begin---
HTTP/1.0 200 OK
Age: 242935
Via: 1.1 eb1a8c1b1275e33a016e623478052111.cloudfront.net (CloudFront)
Date: Tue, 06 Oct 2020 11:28:59 GMT
ETag: "329ffed5681f5f7a792fe31a5716ca6a-19"
Server: AmazonS3
X-Cache: Hit from cloudfront
Connection: Keep-Alive
X-Amz-Cf-Id: 9DWWLDgH9nmtPsySL-J7CnuzHxcQiRqKoUz7rUiAhU3DA3wz5-jAVg==
X-Amz-Cf-Pop: FRA2-C2
Accept-Ranges: bytes
Last-Modified: Mon, 28 Sep 2020 04:45:02 GMT
Content-Length: 986189754
x-amz-version-id: S6sWlyyLr_hVZ8XmNRpyHnVPJsga.v5Z
Content-Disposition: attachment; filename=harbor-container-registry-2.0.3-build.15.pivotal

---response end---
200 OK
Disabling further reuse of socket 4.
Closed 4/SSL 0x0000000000f60cf0
Registered socket 5 for persistent reuse.
Länge: 986189754 (941M)
In »»harbor_v2.0.3.pivotal2«« speichern.

 0% [                                                                                                                                                    ] 986.190      138KB/s  ETA 1h 56m ^
pivotal-ivan-wang commented 4 years ago

Maybe this url needs to be whitelisted? d13k9s5899twdr.cloudfront.net

https://network.pivotal.io/docs/faq#downloading

sluetze commented 4 years ago

Hi, Firewall is open. I can download with wget without problems.

Or do you mean in the proxy for ssl-interception?

bsoroushian commented 3 years ago

Hi,

It seems your proxy changes the status code 206 Partial Content to 200 Ok.

Our downloader relies on 206 to function for multi-chunk download code, and this behaviour is already seen in azure proxies see here.

It can be fixed either by us changing the code for accepting 200 in addition to 206 or you fix your reverse proxy to don't change the status code 206 to 200.

Changing the code on our side won't be easy and we may need a thorough checking for not breaking the normal behaviour.

sluetze commented 3 years ago

Thanks for the Response, i'll reach out to my proxy-team to see what they can do.

bsoroushian commented 3 years ago

@sluetze also if your proxy team indicate that we are missing a header in our response (like Accept-Ranges: bytes) we will appreciate if they indicate in which response your proxy expects to see that header.