Can't install LWP-Simple on older MacOS...

[jubilatious1]

Tried installing LWP::Simple as a dependency of GTK::Simple.

Either alone or as a dependency, zef never completes install of LWP::Simple on older MacOS machine, stalls on 'testing' with 100% CPU utilization:

user@mbook:~$ ~/rakudo/rakudo-2020.10/zef/bin/zef install LWP::Simple 
===> Searching for: LWP::Simple
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Testing: LWP::Simple:ver<0.106>:auth<github:perl6>
^C
~$ ~/rakudo/rakudo-2020.10/zef/bin/zef install LWP::Simple --force
===> Searching for: LWP::Simple
===> Testing: LWP::Simple:ver<0.106>:auth<github:perl6>
^C
user@mbook:~$

[JJ]

We don't test for MacOS enough... Anyway, it would be great if you could run the same with --verbose so that we would see where it's stalling.

[JJ]

Tests now work, but my hunch is that it's somehow not detecting your self-described "older" mac as macosx and falling into bug #28. Can you please check that by running only that test?

[jubilatious1]

I tried to install this morning using the --verbose flag. As the log below shows, the test get-unsized.t was skipped. However the install stalled at the test get-w3-redirect.t with > 95% User CPU utilization. The install was halted (unsuccessfully) after 10 wallclock minutes had passed:

user@mbook:~$ ~/rakudo/rakudo-2020.10/zef/bin/zef install LWP::Simple --verbose
===> Searching for: LWP::Simple
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Found: LWP::Simple:ver<0.106>:auth<github:perl6> [via Zef::Repository::LocalCache]
===> Testing: LWP::Simple:ver<0.106>:auth<github:perl6>
[LWP::Simple] t/000-load-module.t ............. ok
[LWP::Simple] t/basic-auth.t .................. ok
[LWP::Simple] t/custom-headers-and-content.t .. ok
[LWP::Simple] t/delete-headers.t .............. ok
[LWP::Simple] t/get-binary-camelia.t .......... ok
[LWP::Simple] t/get-chunked-6guts.t ........... ok
[LWP::Simple] t/get-headers.t ................. ok
[LWP::Simple] t/get-perl6-org.t ............... ok
[LWP::Simple] t/get-unsized.t ................. skipped: Temporarily disabled on MacOS, see Issue 26
[LWP::Simple] t/get-w3-latin1-utf8.t .......... ok
[LWP::Simple] t/get-w3-redirect.t ............. ok
^C
user@mbook::~$ 

[JJ]

I don't think it's got anything to do with Mac itself. It might be doing some infinite redirects... let me check anyway.

[JJ]

Can you please just try wget http://jigsaw.w3.org/HTTP/300/301.html or do it from the browser to see what happens?

[jubilatious1]

Trying your http://jigsaw.w3.org/HTTP/300/301.html link from a browser (Firefox), it redirects properly to http://jigsaw.w3.org/HTTP/300/Overview.html. (And also, all the other links on the 300 Overview page work properly from Firefox).

Below is wget from the Mac Terminal app:

user@mbook:~$ wget http://jigsaw.w3.org/HTTP/300/301.html 
--2021-01-06 10:18:47--  http://jigsaw.w3.org/HTTP/300/301.html
Resolving jigsaw.w3.org (jigsaw.w3.org)... 128.30.52.21, 2603:400a:ffff:804:801e:34::15
Connecting to jigsaw.w3.org (jigsaw.w3.org)|128.30.52.21|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://jigsaw.w3.org/HTTP/300/Overview.html [following]
--2021-01-06 10:18:48--  http://jigsaw.w3.org/HTTP/300/Overview.html
Reusing existing connection to jigsaw.w3.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 1651 (1.6K) [text/html]
Saving to: ‘301.html’

301.html                                    100%[=========================================================================================>]   1.61K  --.-KB/s    in 0s      

2021-01-06 10:18:48 (112 MB/s) - ‘301.html’ saved [1651/1651]

user@mbook:~$ 

[jubilatious1]

For good measure, repeated the same tests trying to reach https://jigsaw.w3.org/HTTP/300/301.html. Using Firefox I got exactly the same (successful) results as previously.

Below is wget from the Mac Terminal app trying to reach https://jigsaw.w3.org/HTTP/300/301.html (also successful):

user@mbook:~$ wget https://jigsaw.w3.org/HTTP/300/301.html 
--2021-01-06 10:29:03--  https://jigsaw.w3.org/HTTP/300/301.html
Resolving jigsaw.w3.org (jigsaw.w3.org)... 128.30.52.21, 2603:400a:ffff:804:801e:34::15
Connecting to jigsaw.w3.org (jigsaw.w3.org)|128.30.52.21|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://jigsaw.w3.org/HTTP/300/Overview.html [following]
--2021-01-06 10:29:04--  https://jigsaw.w3.org/HTTP/300/Overview.html
Reusing existing connection to jigsaw.w3.org:443.
HTTP request sent, awaiting response... 200 OK
Length: 1651 (1.6K) [text/html]
Saving to: ‘301.html.1’

301.html.1                                  100%[=========================================================================================>]   1.61K  --.-KB/s    in 0s      

2021-01-06 10:29:04 (19.2 MB/s) - ‘301.html.1’ saved [1651/1651]

user@mbook:~$ 

[JJ]

Hum. The thing is, we now test this in MacOSx, as you might have seen, and it does not seem to have a problem. So it's likely there's some combination of older distributions in your system or somesuch. Let me check it anyway.

[JJ]

Here's the last result. Check the OS version in the first item.

[jubilatious1]

I feel that there are a number of Raku users, similar to myself, who are trying out Raku on older machines (out of personal affinity/preference, personal comfort level, or economic realities). I know of one Raku user who uses an older linux box (Ubuntu or Fedora).

The hope here is to amass Raku users with experience on a wide variety of machines, so new users having difficulties have a basis of knowledge to work from. Many open source projects stall because new users are dissuaded from trying out the new open source code (they're told "you have to run the latest OS and/or latest hardware" or something similar).

The MacOS tests you link to are for MacOS Catalina 10.15.7, which was released/updated on September 23, 2020. My MacOS is still a "California name-place OS"--but considerably older.

Is there any other test I can run for you on my end, @JJ? I'd hate to just --force test or --force install and end up with a botched module.

THANK YOU FOR ALL YOUR EFFORTS !!

[JJ]

The problem is that it's complicated to support that it's not available for testing. Same happens with FreeBSD, spent the whole day trying to set it up to no avail. (But thanks to @Tyil I might have a break now) I'll try and see if there's support for that somewhere, but you can help meanwhile trying to find if there's an in infinite loop right here in this module or some upstream dependency, which is far more likely

[jubilatious1]

Is the problem with https://github.com/raku-community-modules/LWP-Simple/blob/master/t/getstore.t ??

If you look at the --verbose output, the testing line [LWP::Simple] t/get-w3-redirect.t ............. ok returns OK.

The next test on the list is getstore.t.

FYI I'm on Rakudo_2020.10. And if you need me to, I can try tests on older installed versions of Rakudo. In fact, my logs indicate that I had successfully installed LWP::Simple under Rakudo_2020.02.1.

HTH.

[jubilatious1]

The [LWP::Simple] t/get-w3-redirect.t result looks fine (ran test from downloaded rakudo-star-2020.05 src):

user@mbook:~$ raku /Users/me/rakudo/rakudo-star-2020.05/src/rakudo-star-modules/LWP-Simple/t/get-w3-redirect.t
1..1
ok 1 - Was redirected to w3 redirect test page
user@mbook:~$ raku --version
Welcome to 𝐑𝐚𝐤𝐮𝐝𝐨™ v2020.10.
Implementing the 𝐑𝐚𝐤𝐮™ programming language v6.d.
Built on MoarVM version 2020.10.

[jubilatious1]

Running the [LWP::Simple] t/getstore.t from downloaded rakudo-star-2020.05 src results in a hang, with the error "error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure":

user@mbook:~$ raku /Users/me/rakudo/rakudo-star-2020.05/src/rakudo-star-modules/LWP-Simple/t/getstore.t
1..10
ok 1 - getstore() returned success
ok 2 - Opened file handle written by getstore()
ok 3 - Found pattern in downloaded file
ok 4 - Close the temporary file
ok 5 - Delete the temporary file
err code: 336032784
error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
^C
user@mbook:~$ raku --version
Welcome to 𝐑𝐚𝐤𝐮𝐝𝐨™ v2020.10.
Implementing the 𝐑𝐚𝐤𝐮™ programming language v6.d.
Built on MoarVM version 2020.10.

A quick perusal of StackOverflow turns up this exact same error code for PHP/cURL:

https://stackoverflow.com/questions/38794133/errorerror14077410ssl-routinesssl23-get-server-hellosslv3-alert-handshake-f

The suggested course of action for PHP/cURL is to force TLS 1.2 with the cURL option curl_setopt($ch, CURLOPT_SSLVERSION, 6); . Since this is LWP I was able to find some Perl(5) LWP info here:

https://stackoverflow.com/questions/49166769/using-perl-5-8-8-with-openssl-1-0-1-to-connect-with-tlsv1-2/49166945#49166945

The SO question above regards trying to use OpenSSL_1.0.1 / Perl5 LWP / TLS1.2 together. Since my MacOS system is already at OpenSSL 1.1.1g I think this is fixable issue in the Raku LWP::Simple module itself.

I'll try to read over those Perl5 LWP references in more depth. Thanks again @JJ for all your help!

[JJ]

But that's a problem with the IO::Socket::SSL library that was eventually fixed. I don't think it's got anything to do with the test stalling you described before. I'll try to get this tested for more versions of... Well, everything. Thanks for your help anyway.

[jubilatious1]

I'm at IO::Socket::SSL:ver<0.0.2> . Maybe @sergot has some ideas?

user@mbook:~$ ~/rakudo/rakudo-2020.10/zef/bin/zef upgrade IO::Socket::SSL --debug
===> Searching for: IO::Socket::SSL
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Found: IO::Socket::SSL:ver<0.0.2>:auth<github:sergot> [via Zef::Repository::LocalCache]
All requested distributions are already at their latest versions
user@mbook:~$ 

[JJ]

So it's not that. Let me see if I can have a look at this today, but holidays are finished, so...

[JJ]

I'm checking out and Mojave is available in Appveyor, Travis goes down to 10.11, but I don't have the privs to install it in this organization (and I don't really know who does). So I'm afraid you will need to peel the layers of dependencies, to see where's the problem and upgrade as needed. This module anyway is a very thin layer around IO::Socket::Inet, so it's likely that the problem is there (and is not caught because untested).

[jubilatious1]

Hi @JJ ,

I can successfully get past the t/getstore.t test with the recent pull request. However I now seem to be stalling at t/issue-7.t , which tests against access to http:/github.com .

I think this is a pure SSL issue so I'm looking at IO::Socket::SSL . It seems that http://github.com requires TLS1.2 which I don't know how to set with LWP::Simple , IO::Socket::SSL , or an environmental variable in my OS.

The short answer is that it's fine to close this issue. I'll keep working on t/issue-7.t to see if I get anywhere.

Thank you!

user@mbook:~$ ~/rakudo/rakudo-2020.10/zef/bin/zef install LWP::Simple --debug
===> Searching for: LWP::Simple
===> Updating cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Updating p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated p6c mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/p6c1.json
===> Updated cpan mirror: https://raw.githubusercontent.com/ugexe/Perl6-ecosystems/master/cpan1.json
===> Found: LWP::Simple:ver<0.107>:auth<github:raku-community-modules> [via Zef::Repository::LocalCache]
===> Dependencies: MIME::Base64, URI, JSON::Tiny
===> Filtering: LWP::Simple:ver<0.107>:auth<github:raku-community-modules>
===> Filtering [OK] for LWP::Simple:ver<0.107>:auth<github:raku-community-modules>
===> # SKIP: No need to build LWP::Simple:ver<0.107>:auth<github:raku-community-modules>
===> Testing: LWP::Simple:ver<0.107>:auth<github:raku-community-modules>
[LWP::Simple] Testing with plugin: Zef::Service::Shell::prove+{<anon|1>}
[LWP::Simple] t/000-load-module.t ............. ok
[LWP::Simple] t/basic-auth.t .................. ok
[LWP::Simple] t/custom-headers-and-content.t .. ok
[LWP::Simple] t/delete-headers.t .............. ok
[LWP::Simple] t/get-binary-camelia.t .......... ok
[LWP::Simple] t/get-chunked-6guts.t ........... ok
[LWP::Simple] t/get-headers.t ................. ok
[LWP::Simple] t/get-perl6-org.t ............... ok
[LWP::Simple] t/get-unsized.t ................. ok
[LWP::Simple] t/get-w3-latin1-utf8.t .......... ok
[LWP::Simple] t/get-w3-redirect.t ............. ok
[LWP::Simple] t/getstore.t .................... ok
[LWP::Simple] t/head.t ........................ ok
^C
user@mbook:~$ 

[JJ]

Maybe we can switch back google.com to the original URL? It was created for test purposes, so it should be able to use another version of TLS (or none)

[jubilatious1]

My system was hanging on t/getstore.t , so I changed the old http test URL (which I think was http://eu.httpbin.org/anything/Web) to http://www.google.com .

My reasoning is 1) it seems more robust and 2) I wasn't sure about the OS/browser performing some sort of "Insecure Request Upgrade" (to https). Certainly when I checked with my Firefox browser it seems a http request gets upgraded to https via the "Upgrade-Insecure-Requests" setting.

My guess is the problem was/is with SSL for the https request. So if the http request was getting upgraded to https I didn't want to fail 10 out of 10 tests, I wanted to fail only the last 5 out of 10 tests.

So I guess I have to kick the question back to you: what does the t/getstore.t test test? When it is successful it executes the code ok($fh.slurp ~~ $rx, ...) and returns the string 'Found pattern in downloaded file'. It doesn't say anything more than that. But if it's supposed to be doing more, (like a get request), then it's entirely up to you what you want the http URL to be.

OTOH, I don't see any way to (or any reason to) revert the new https URL of https://openssl.org because it seems to be the crux of the fix (and it's unlikely that https://openssl.org will have misconfigured SSL on their own website).

Best Regards!

[JJ]

But in this case I think it would be better to leave it at http... It's testing a different thing then... And I really have no idea what it tests. I really have to go through this... It might have been tested elsewhere. But if it's hanging, it's probably not, and it's testing something valuable.

[jubilatious1]

I'd love @sergot to take a peek at this, since Raku's LWP-Simple depends on his IO::Socket::SSL module.

Or maybe someone intimately familiar with Perl5 LWP-Simple, like @oalders (also probably knowledgable in SSL)?

Or even a general SSL expert who might have a latent interest in Raku (a.k.a. Perl6), someone like @noxxi ?

[noxxi]

Since I got mentioned here ... I'm not familiar with Raku. But the symptoms look for me like a problem with a too old version of OpenSSL, specifically missing support for TLS 1.2.

MacOS came for many years with OpenSSL 0.9.8 only, which did not support TLS 1.2. It switched with MacOS 10.13 in 2017 (timing information based on this) to LibreSSL, which was forked from OpenSSL 1.0.2 and supports TLS 1.2.

While @jubilatious1 claims that OpenSSL 1.1.1 is installed on the system, there is a difference between installed and actually used. Typically new installations of OpenSSL (like with homebrew or self-compiled) don't actually replace the existing OpenSSL version and its include files, but instead just add the new version into a new path and have this path be first in PATH so that it gets executed instead of the older version. This has no effect on the path used for building from source code though. Here the standard include and library path will be used unless explicitly set differently.

This means it will on these old platforms build by default with OpenSSL 0.9.8, no matter if newer versions would be available too. And then it will not be able to use TLS 1.2, no matter if trying to explicitly set this or not within LWP - support is simply not in the used library. This is actually a repeated problem seen in questions on StackOverflow when building Python on these older MacOS versions. I'm pretty sure that using something like dtruss to actually see what files are used when running the program would show the use of the older OpenSSL library instead of the newer one.

I'm not familiar on how the OpenSSL bindings actually work in Raku. But somewhere the correct include and library path must be given when building the modules, so that it will actually use the newly installed version of OpenSSL in the non-standard path.

[JJ]

jigsaw is simply not working. #49 is another example. Will address both.