search

sergot / http-useragent

Web user agent class for Perl 6.
MIT License
36 stars 39 forks source link

Fix #229 #230

Closed pheix closed 4 years ago

pheix commented 4 years ago

Test ./t/150-issue-64.t fails with NETWORK_TESTING env var due to $.useragent is not setup. It seems that https://www.dublincore.org/ rejects requests without UserAgent header field.

jonathanstowe commented 4 years ago

Hahaha, the reason that test existed in the first place was because that host was doing something unusual with redirects IIRC.

pheix commented 4 years ago

Hahaha, the reason that test existed in the first place was because that host was doing something unusual with redirects IIRC.

As I see the host makes a queue of redirects. If add $response.code.say; to the top of this condition block we've got:

[kostas@webtech-macbook http-useragent]$ perl6 -Ilib ./t/150-issue-64.t 
1..3
302
303
301
ok 1 - make request to 'http://purl.org/dc/elements/1.1/' lives

  1. first 302 redirect follows to https://dublincore.org/specifications/dublin-core/dcmi-terms/dublin_core_elements

  2. second 303 redirect follows to http://dublincore.org/specifications/dublin-core/dcmi-terms/ (no HTTPS)

  3. third 301 redirect follows to https://www.dublincore.org/specifications/dublin-core/dcmi-terms/ (with HTTPS)

IMHO, there's no something unusual, except 303 redirect code at second step (303 redirect is used as a response to POST/PUT request).

But what is really interesting and unusual: https://dublincore.org/specifications/dublin-core/dcmi-terms/dublin_core_elements falls to 404 with no useragent (or non-standard UA) requests:

[kostas@webtech-macbook http-useragent]$ curl -A "" https://dublincore.org/specifications/dublin-core/dcmi-terms/dublin_core_elements
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at dublincore.org Port 443</address>
</body></html>
[kostas@webtech-macbook http-useragent]$

and 

[kostas@webtech-macbook http-useragent]$ curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:14.0) Gecko/20120405 Firefox/14.0a1" https://dublincore.org/specifications/dublin-core/dcmi-terms/dublin_core_elements
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>303 See Other</title>
</head><body>
<h1>See Other</h1>
<p>The answer to your request is located <a href="http://dublincore.org/specifications/dublin-core/dcmi-terms/">here</a>.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at dublincore.org Port 443</address>
</body></html>
[kostas@webtech-macbook http-useragent]$