BUG: acbuild ignores proxy

[Andrei-Pozolotin]

http.ProxyFromEnvironment

see https://github.com/appc/acbuild/blob/master/registry/fetch.go#L357

compared to https://github.com/appc/spec/blob/master/discovery/http.go#L58

[cgonyeo]

It looks like the http.DefaultTransport sets the proxy to be ProxyFromEnvironment, so I should just need to do that when the insecure flag is set and I don't use the default transport. I'll make a PR.

[cgonyeo]

@Andrei-Pozolotin any chance you'd be willing to see if https://github.com/appc/acbuild/pull/239 fixes this for you?

[Andrei-Pozolotin]

@dgonyeo

1) thank you for the PR

2) test result: it does partially resolve the issue

3) remaining problem is: Proxy: http.ProxyFromEnvironment should be applied regardless of --insecure / r.Insecure, in other words: "proxy always applies" https://github.com/appc/acbuild/blob/master/registry/fetch.go#L363

[Andrei-Pozolotin]

1) another issue came up, which has to do with quay.io servers,

2) for example, a portion of build script:

acbuild dep add quay.io/coreos/alpine-sh

3) produces the following proxy log:

2016-Jul-18 20:47:16    211 192.168.1.103 TAG_NONE/200 0 CONNECT quay.io:443 - HIER_DIRECT/174.129.232.79 -
2016-Jul-18 20:47:16    123 192.168.1.103 TCP_MISS/302 719 GET https://quay.io/coreos/alpine-sh?ac-discovery=1 - HIER_DIRECT/174.129.232.79 text/html
2016-Jul-18 20:47:16     79 192.168.1.103 TCP_MISS/200 17961 GET https://quay.io/repository/coreos/alpine-sh?tag=latest - HIER_DIRECT/174.129.232.79 text/html
2016-Jul-18 20:47:16    144 192.168.1.103 TAG_NONE/200 0 CONNECT quay.io:443 - HIER_DIRECT/174.129.232.79 -
2016-Jul-18 20:47:16    134 192.168.1.103 TCP_MISS/302 1127 GET https://quay.io/c1/aci/quay.io/coreos/alpine-sh/latest/aci/linux/amd64/ - HIER_DIRECT/174.129.232.79 text/html
2016-Jul-18 20:47:16    174 192.168.1.103 TAG_NONE/200 0 CONNECT quay-registry.s3.amazonaws.com:443 - HIER_DIRECT/54.231.113.219 -
2016-Jul-18 20:47:18   1174 192.168.1.103 TCP_MISS/200 2652298 GET https://quay-registry.s3.amazonaws.com/sharedimages/5fed0187-c814-4eee-b340-cc7f26230872/layer?Signature=QZAkBUt4ly%2FDsHbikHuGMrNQcAI%3D&Expires=1468892898&AWSAccessKeyI

4) with response headers like:

< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Tue, 19 Jul 2016 02:32:28 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 78998
< X-FRAME-OPTIONS: DENY
< Cache-Control: no-cache, no-store, must-revalidate
< Set-Cookie: session=eyJfY3NyZl90b2tlbiI6eyIgYiI6IlpIRnpaRUZ1SzNGa2JUbHdNa2xHUWpOYVdtVklORmRMVG5wc1luWjFTVFl2ZFhGaEwySkdXRXQ0YkM4cmRIVm5lUzlFWmpaVFMzTlpTM0pTTm1SbVNRPT0ifX0.Cm8ivA.3I7V-56NSwDrPk_mgIQy95qH2ec; Secure; HttpOnly; Path=/
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=63072000; preload
< X-Cache: MISS from proxy
< X-Cache-Lookup: MISS from proxy:3128
< Connection: keep-alive

5) where

Cache-Control: no-cache, no-store, must-revalidate

means that quay/amazon produces transient resources, and makes proxy useless for the quay.io images

6) this happens for both tag=latest and tag=some-version-number

7) surely we can hack around this by header override on the proxy, but is it a feature or a bug?

[Andrei-Pozolotin]

re: quay.io servers:

one can make it work with squid 3.5 / store_id: http://wiki.squid-cache.org/Features/StoreID

here is minimal working config https://gist.github.com/Andrei-Pozolotin/0b65b5ab5813dca94886c1ebe9158a18

probably should document proxy setup for all sites: acbuild, rkt, quay.io

[cgonyeo]

remaining problem is: Proxy: http.ProxyFromEnvironment should be applied regardless of --insecure / r.Insecure, in other words: "proxy always applies"

Did what I wrote not work without the insecure flag? Weird, because the http.DefaultTransport does set http.ProxyFromEnvironment: https://golang.org/src/net/http/transport.go

I've pushed a change that always sets it now.

Also I'd be nervous about overriding headers, so probably better to just document how to setup a proxy for this stuff. As it touches rkt, acbuild, and quay.io, I'm not entirely certain where it should go. Adding it to acbuild is a start though, so if you don't mind I could at the very least pull in your minimal configs.

I'm also not familiar with squid at all myself, so if you have the time to write a short markdown thing explaining it I'd love to have that in acbuild docs.

[Andrei-Pozolotin]

I've pushed a change that always sets it now.

your change works, thank you

because the http.DefaultTransport does set http.ProxyFromEnvironment

not sure why this is needed, probably should comment code that proxy needs extra testing

not familiar with squid at all myself, so if you have the time

I can contribute to the following: how about if you setup public demo build project somewhere? which would need to: 1) spawn squid container, then 2) spawn build container to run acbuild against quay.io via squid 3) spawn runtime container to run rkt against that image via squid then result would be self documenting on every stage and documentation will go in relevant configuration files