search

libwww-perl / LWP-Protocol-https

Provide https support for LWP::UserAgent
https://metacpan.org/pod/LWP::Protocol::https
Other
16 stars 35 forks source link

Tests fail with HTTPS_PROXY set [rt.cpan.org #117479] #39

Closed oalders closed 3 years ago

oalders commented 7 years ago

Migrated from rt.cpan.org#117479 (status was 'new')

Requestors:

From $_ = 'spro^^%^6ut#@&$>#!^!#&!pan.org'; y/a-z.@//cd; print on 2016-09-04 01:26:14:

dakkar commented 3 years ago

the import from RT didn't quite work…

the original ticket said:

LWP::Protocol::https version 6.07.

LWP’s env_proxy appears to prefer HTTPS_PROXY to https_proxy, so t/https_proxy.t’s https_proxy env var setting gets ignored:

t/https_proxy.t .. 1/56 unexpected response: HTTP/1.1 403 Forbidden Connection: close

and provided a patch (I'm attaching it)

dakkar commented 3 years ago 
diff -rup LWP-Protocol-https-6.07-1/t/https_proxy.t LWP-Protocol-https-6.07-0/t/https_proxy.t
--- LWP-Protocol-https-6.07-1/t/https_proxy.t   2016-06-30 03:43:17.000000000 -0700
+++ LWP-Protocol-https-6.07-0/t/https_proxy.t   2017-10-30 22:13:29.000000000 -0700
@@ -85,6 +85,7 @@ $ua{proxy_nokeepalive} = LWP::UserAgent-
    SSL_ca_file => $cafile
     }
 );
+$ENV{HTTP_PROXY} = $ENV{HTTPS_PROXY} =
 $ENV{http_proxy} = $ENV{https_proxy} = "http://foo:bar\@$saddr[0]";
 $ua{proxy}->env_proxy;
 $ua{proxy_nokeepalive}->env_proxy;
dakkar commented 3 years ago

possibly separate: even when those env variables have empty values, they still affect the module's operation, which is a bit annoying when trying to run something without proxy, because http_proxy= https_proxy= HTTP_PROXY= HTTPS_PROXY= the_program still tries to talk to a non-existent proxy

oalders commented 3 years ago

Looks to be related to https://github.com/libwww-perl/libwww-perl/issues/230

oalders commented 3 years ago

The proposed fix is at https://github.com/gisle/libwww-perl/pull/3

oalders commented 3 years ago

See also https://github.com/libwww-perl/libwww-perl/commit/a87c28123254c9b38be

dakkar commented 3 years ago

all of that looks very related, yes. I'm not sure how much of those fixes have been released, though.

This happens, with LWP-Protocol-https-6.09 (sometimes, randomly):

]$ HTTPS_PROXY= prove -lv t/https_proxy.t 
t/https_proxy.t .. 
1..56
ok 1 - noproxy http://127.0.0.1:57597/foo -> A.1@nossl
ok 2 - URL in request -> /foo
ok 3 - noproxy http://127.0.0.1:57597/bar -> A.2@nossl
ok 4 - URL in request -> /bar
ok 5 - noproxy http://127.0.0.1:37563/foo -> B.1@nossl
ok 6 - URL in request -> /foo
ok 7 - noproxy http://127.0.0.1:37563/bar -> B.2@nossl
ok 8 - URL in request -> /bar
ok 9 - noproxy http://127.0.0.1:57597/tor -> A.3@nossl
ok 10 - URL in request -> /tor
ok 11 - noproxy http://127.0.0.1:37563/tor -> B.3@nossl
ok 12 - URL in request -> /tor
ok 13 - proxy http://foo/foo -> C.1.auth@nossl
ok 14 - URL in request -> http://foo/foo
ok 15 - proxy http://foo/bar -> C.2.auth@nossl
ok 16 - URL in request -> http://foo/bar
ok 17 - proxy http://bar/foo -> C.3.auth@nossl
ok 18 - URL in request -> http://bar/foo
ok 19 - proxy http://bar/bar -> C.4.auth@nossl
ok 20 - URL in request -> http://bar/bar
ok 21 - proxy http://foo/tor -> C.5.auth@nossl
ok 22 - URL in request -> http://foo/tor
ok 23 - proxy http://bar/tor -> C.6.auth@nossl
ok 24 - URL in request -> http://bar/tor
# creating cert for direct.ssl.access
ok 25 - noproxy https://127.0.0.1:57597/foo -> D.1@direct.ssl.access
ok 26 - URL in request -> /foo
ok 27 - noproxy https://127.0.0.1:57597/bar -> D.2@direct.ssl.access
ok 28 - URL in request -> /bar
# creating cert for direct.ssl.access
ok 29 - noproxy https://127.0.0.1:37563/foo -> E.1@direct.ssl.access
ok 30 - URL in request -> /foo
ok 31 - noproxy https://127.0.0.1:37563/bar -> E.2@direct.ssl.access
ok 32 - URL in request -> /bar
ok 33 - noproxy https://127.0.0.1:57597/tor -> D.3@direct.ssl.access
ok 34 - URL in request -> /tor
ok 35 - noproxy https://127.0.0.1:37563/tor -> E.3@direct.ssl.access
ok 36 - URL in request -> /tor
unexpected response: 500 Can't connect to foo:443 (Name or service not known)
Content-Type: text/plain
Client-Date: Mon, 14 Dec 2020 18:16:02 GMT
Client-Warning: Internal response

Can't connect to foo:443 (Name or service not known)

Name or service not known at /home/dakkar/perl5/perlbrew/perls/perl-5.32.0/lib/site_perl/5.32.0/LWP/Protocol/http.pm line 50.
# Looks like your test exited with 22 just after 36.
Dubious, test returned 22 (wstat 5632, 0x1600)
Failed 20/56 subtests 

Test Summary Report
-------------------
t/https_proxy.t (Wstat: 5632 Tests: 36 Failed: 0)
  Non-zero exit status: 22
  Parse errors: Bad plan.  You planned 56 tests but ran 36.
Files=1, Tests=36,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.20 cusr  0.02 csys =  0.25 CPU)
Result: FAIL

I see two problems:

  1. the hash-order / conflicting env vars as noted in the other issues/PRs
  2. an empty env variable is not ignored, but I think it should
oalders commented 3 years ago

@dakkar are you able to test if the random failures disappear with gisle/libwww-perl#3?

dakkar commented 3 years ago

Something is different. I checked out https://github.com/demerphq/libwww-perl at that commit, then:

$ HTTP_PROXY= HTTPS_PROXY= PERL5LIB=/tmp/libwww-perl/lib prove -lv t/https_proxy.t 
t/https_proxy.t .. 
1..56
ok 1 - noproxy http://127.0.0.1:48887/foo -> A.1@nossl
ok 2 - URL in request -> /foo
ok 3 - noproxy http://127.0.0.1:48887/bar -> A.2@nossl
ok 4 - URL in request -> /bar
ok 5 - noproxy http://127.0.0.1:47551/foo -> B.1@nossl
ok 6 - URL in request -> /foo
ok 7 - noproxy http://127.0.0.1:47551/bar -> B.2@nossl
ok 8 - URL in request -> /bar
ok 9 - noproxy http://127.0.0.1:48887/tor -> A.3@nossl
ok 10 - URL in request -> /tor
ok 11 - noproxy http://127.0.0.1:47551/tor -> B.3@nossl
ok 12 - URL in request -> /tor
ok 13 - proxy http://foo/foo -> C.1.auth@nossl
ok 14 - URL in request -> http://foo/foo
ok 15 - proxy http://foo/bar -> C.2.auth@nossl
ok 16 - URL in request -> http://foo/bar
ok 17 - proxy http://bar/foo -> C.3.auth@nossl
ok 18 - URL in request -> http://bar/foo
ok 19 - proxy http://bar/bar -> C.4.auth@nossl
ok 20 - URL in request -> http://bar/bar
ok 21 - proxy http://foo/tor -> C.5.auth@nossl
ok 22 - URL in request -> http://foo/tor
ok 23 - proxy http://bar/tor -> C.6.auth@nossl
ok 24 - URL in request -> http://bar/tor
# creating cert for direct.ssl.access
ok 25 - noproxy https://127.0.0.1:48887/foo -> D.1@direct.ssl.access
ok 26 - URL in request -> /foo
ok 27 - noproxy https://127.0.0.1:48887/bar -> D.2@direct.ssl.access
ok 28 - URL in request -> /bar
# creating cert for direct.ssl.access
ok 29 - noproxy https://127.0.0.1:47551/foo -> E.1@direct.ssl.access
ok 30 - URL in request -> /foo
ok 31 - noproxy https://127.0.0.1:47551/bar -> E.2@direct.ssl.access
ok 32 - URL in request -> /bar
ok 33 - noproxy https://127.0.0.1:48887/tor -> D.3@direct.ssl.access
ok 34 - URL in request -> /tor
ok 35 - noproxy https://127.0.0.1:47551/tor -> E.3@direct.ssl.access
ok 36 - URL in request -> /tor
not ok 37 - proxy https://foo/foo -> C.8.Tauth@foo

#   Failed test 'proxy https://foo/foo -> C.8.Tauth@foo'
#   at t/https_proxy.t line 182.
#          got: 'C.7.auth@nossl'
#     expected: 'C.8.Tauth@foo'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 181
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 7
# 
# ID: 3.7.auth@nossl
# ---------
# GET https://foo/foo HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: foo
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 38 - URL in request -> /foo

#   Failed test 'URL in request -> /foo'
#   at t/https_proxy.t line 190.
#          got: 'https://foo/foo'
#     expected: '/foo'
not ok 39 - proxy https://foo/bar -> C.9.Tauth@foo

#   Failed test 'proxy https://foo/bar -> C.9.Tauth@foo'
#   at t/https_proxy.t line 182.
#          got: 'C.8.auth@nossl'
#     expected: 'C.9.Tauth@foo'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 181
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 8
# 
# ID: 3.8.auth@nossl
# ---------
# GET https://foo/bar HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: foo
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 40 - URL in request -> /bar

#   Failed test 'URL in request -> /bar'
#   at t/https_proxy.t line 190.
#          got: 'https://foo/bar'
#     expected: '/bar'
not ok 41 - proxy https://bar/foo -> F.2.Tauth@bar

#   Failed test 'proxy https://bar/foo -> F.2.Tauth@bar'
#   at t/https_proxy.t line 182.
#          got: 'C.9.auth@nossl'
#     expected: 'F.2.Tauth@bar'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 181
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 9
# 
# ID: 3.9.auth@nossl
# ---------
# GET https://bar/foo HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: bar
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 42 - URL in request -> /foo

#   Failed test 'URL in request -> /foo'
#   at t/https_proxy.t line 190.
#          got: 'https://bar/foo'
#     expected: '/foo'
not ok 43 - proxy https://bar/bar -> F.3.Tauth@bar

#   Failed test 'proxy https://bar/bar -> F.3.Tauth@bar'
#   at t/https_proxy.t line 182.
#          got: 'C.10.auth@nossl'
#     expected: 'F.3.Tauth@bar'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 182
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 10
# 
# ID: 3.10.auth@nossl
# ---------
# GET https://bar/bar HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: bar
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 44 - URL in request -> /bar

#   Failed test 'URL in request -> /bar'
#   at t/https_proxy.t line 190.
#          got: 'https://bar/bar'
#     expected: '/bar'
not ok 45 - proxy https://foo/tor -> C.10.Tauth@foo

#   Failed test 'proxy https://foo/tor -> C.10.Tauth@foo'
#   at t/https_proxy.t line 182.
#          got: 'C.11.auth@nossl'
#     expected: 'C.10.Tauth@foo'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 182
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 11
# 
# ID: 3.11.auth@nossl
# ---------
# GET https://foo/tor HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: foo
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 46 - URL in request -> /tor

#   Failed test 'URL in request -> /tor'
#   at t/https_proxy.t line 190.
#          got: 'https://foo/tor'
#     expected: '/tor'
not ok 47 - proxy https://bar/tor -> F.4.Tauth@bar

#   Failed test 'proxy https://bar/tor -> F.4.Tauth@bar'
#   at t/https_proxy.t line 182.
#          got: 'C.12.auth@nossl'
#     expected: 'F.4.Tauth@bar'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 182
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 12
# 
# ID: 3.12.auth@nossl
# ---------
# GET https://bar/tor HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE
# Host: bar
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 48 - URL in request -> /tor

#   Failed test 'URL in request -> /tor'
#   at t/https_proxy.t line 190.
#          got: 'https://bar/tor'
#     expected: '/tor'
not ok 49 - proxy_nokeepalive https://foo/foo -> H.2.Tauth@foo

#   Failed test 'proxy_nokeepalive https://foo/foo -> H.2.Tauth@foo'
#   at t/https_proxy.t line 182.
#          got: 'H.1.auth@nossl'
#     expected: 'H.2.Tauth@foo'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 188
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 1
# 
# ID: 6.1.auth@nossl
# ---------
# GET https://foo/foo HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE, close
# Host: foo
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 50 - URL in request -> /foo

#   Failed test 'URL in request -> /foo'
#   at t/https_proxy.t line 190.
#          got: 'https://foo/foo'
#     expected: '/foo'
not ok 51 - proxy_nokeepalive https://foo/bar -> I.2.Tauth@foo

#   Failed test 'proxy_nokeepalive https://foo/bar -> I.2.Tauth@foo'
#   at t/https_proxy.t line 182.
#          got: 'I.1.auth@nossl'
#     expected: 'I.2.Tauth@foo'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 188
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 1
# 
# ID: 7.1.auth@nossl
# ---------
# GET https://foo/bar HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE, close
# Host: foo
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 52 - URL in request -> /bar

#   Failed test 'URL in request -> /bar'
#   at t/https_proxy.t line 190.
#          got: 'https://foo/bar'
#     expected: '/bar'
not ok 53 - proxy_nokeepalive https://bar/foo -> J.2.Tauth@bar

#   Failed test 'proxy_nokeepalive https://bar/foo -> J.2.Tauth@bar'
#   at t/https_proxy.t line 182.
#          got: 'J.1.auth@nossl'
#     expected: 'J.2.Tauth@bar'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 188
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 1
# 
# ID: 8.1.auth@nossl
# ---------
# GET https://bar/foo HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE, close
# Host: bar
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 54 - URL in request -> /foo

#   Failed test 'URL in request -> /foo'
#   at t/https_proxy.t line 190.
#          got: 'https://bar/foo'
#     expected: '/foo'
not ok 55 - proxy_nokeepalive https://bar/bar -> K.2.Tauth@bar

#   Failed test 'proxy_nokeepalive https://bar/bar -> K.2.Tauth@bar'
#   at t/https_proxy.t line 182.
#          got: 'K.1.auth@nossl'
#     expected: 'K.2.Tauth@bar'
# HTTP/1.1 200 ok
# Connection: keep-alive
# Content-Length: 188
# Content-Type: text/plain
# Client-Date: Mon, 14 Dec 2020 19:15:56 GMT
# Client-Peer: 127.0.0.1:48887
# Client-Response-Num: 1
# 
# ID: 9.1.auth@nossl
# ---------
# GET https://bar/bar HTTP/1.1
# TE: deflate,gzip;q=0.3
# Connection: TE, close
# Host: bar
# Proxy-Authorization: Basic Zm9vOmJhcg==
# User-Agent: libwww-perl/6.05
not ok 56 - URL in request -> /bar

#   Failed test 'URL in request -> /bar'
#   at t/https_proxy.t line 190.
#          got: 'https://bar/bar'
#     expected: '/bar'
# Looks like you failed 20 tests of 56.
Dubious, test returned 20 (wstat 5120, 0x1400)
Failed 20/56 subtests 

Test Summary Report
-------------------
t/https_proxy.t (Wstat: 5120 Tests: 56 Failed: 20)
  Failed tests:  37-56
  Non-zero exit status: 20
Files=1, Tests=56,  0 wallclock secs ( 0.04 usr  0.00 sys +  0.27 cusr  0.02 csys =  0.33 CPU)
Result: FAIL
dakkar commented 3 years ago

oh, should have added: setting HTTP_PROXY and HTTPS_PROXY to empty strings, or not setting them at all, produces the same test result.

But:

$ HTTP_PROXY=a HTTPS_PROXY=b PERL5LIB=/tmp/libwww-perl/lib prove -lv t/https_proxy.t 
t/https_proxy.t .. Proxy must be specified as absolute URI; 'b' is not at t/https_proxy.t line 89.

and

$ HTTP_PROXY=http://localhost HTTPS_PROXY=https://localhost PERL5LIB=/tmp/libwww-perl/lib prove -lv t/https_proxy.t 
t/https_proxy.t .. Environment contains multiple differing definitions for 'http_proxy'.
Using value from 'HTTP_PROXY' (http://localhost) and ignoring 'http_proxy' (http://foo:bar@127.0.0.1:36237) at /tmp/libwww-perl/lib/LWP/UserAgent.pm line 1017.
Environment contains multiple differing definitions for 'https_proxy'.
Using value from 'HTTPS_PROXY' (https://localhost) and ignoring 'https_proxy' (http://foo:bar@127.0.0.1:36237) at /tmp/libwww-perl/lib/LWP/UserAgent.pm line 1017.
Environment contains multiple differing definitions for 'http_proxy'.
Using value from 'HTTP_PROXY' (http://localhost) and ignoring 'http_proxy' (http://foo:bar@127.0.0.1:36237) at /tmp/libwww-perl/lib/LWP/UserAgent.pm line 1017.
Environment contains multiple differing definitions for 'https_proxy'.
Using value from 'HTTPS_PROXY' (https://localhost) and ignoring 'https_proxy' (http://foo:bar@127.0.0.1:36237) at /tmp/libwww-perl/lib/LWP/UserAgent.pm line 1017.

1..56
ok 1 - noproxy http://127.0.0.1:36237/foo -> A.1@nossl
ok 2 - URL in request -> /foo
ok 3 - noproxy http://127.0.0.1:36237/bar -> A.2@nossl
ok 4 - URL in request -> /bar
ok 5 - noproxy http://127.0.0.1:38743/foo -> B.1@nossl
ok 6 - URL in request -> /foo
ok 7 - noproxy http://127.0.0.1:38743/bar -> B.2@nossl
ok 8 - URL in request -> /bar
ok 9 - noproxy http://127.0.0.1:36237/tor -> A.3@nossl
ok 10 - URL in request -> /tor
ok 11 - noproxy http://127.0.0.1:38743/tor -> B.3@nossl
ok 12 - URL in request -> /tor
unexpected response: 500 Can't connect to localhost:80
Content-Type: text/plain
Client-Date: Mon, 14 Dec 2020 19:36:47 GMT
Client-Warning: Internal response

Can't connect to localhost:80

Connection refused at /tmp/libwww-perl/lib/LWP/Protocol/http.pm line 51.
# Looks like your test exited with 111 just after 12.
Dubious, test returned 111 (wstat 28416, 0x6f00)
Failed 44/56 subtests 

Test Summary Report
-------------------
t/https_proxy.t (Wstat: 28416 Tests: 12 Failed: 0)
  Non-zero exit status: 111
  Parse errors: Bad plan.  You planned 56 tests but ran 12.
Files=1, Tests=12,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.26 cusr  0.01 csys =  0.30 CPU)
Result: FAIL

reliably

oalders commented 3 years ago

Thanks @dakkar! That's helpful information. Maybe the first step would be getting that PR to work with the latest version of libwww-perl?

dakkar commented 3 years ago

I'd say so, yes

oalders commented 3 years ago

Ok, I'll take a quick look at it now.

oalders commented 3 years ago

Rebased PR is at https://github.com/libwww-perl/libwww-perl/pull/355

oalders commented 3 years ago

@dakkar the PR above has now been merged. Can you try HTTP_PROXY= HTTPS_PROXY= PERL5LIB=/tmp/libwww-perl/lib prove -lv t/https_proxy.t again using the updated repository? On my system the tests are passing.

dakkar commented 3 years ago

confirm all passing!

dakkar commented 3 years ago

notice though, that HTTP_PROXY=http://localhost HTTPS_PROXY=http://localhost PERL5LIB=/tmp/libwww-perl/lib prove -l t/https_proxy.t still fails: the test should set/clear those env vars

oalders commented 3 years ago

@dakkar are you able to put together a pull request for this?