need option to pass option to "don't verify SSL" to underlying download method (`wget`, `curl`, _lwp_)

[wwlwpd]

I have to use --mirror to point to https://www.cpan.org, but because of the letsencryptolypse recently, I am needing a way to tell the underlying download methods to not validate SSL.

• wget --no-check-certificate
• curl -k or curl --insecure
• no idea what to use for lwp

  
  cpanm --verbose --mirror https://www.cpan.org Archive::Zip
  cpanm (App::cpanminus) 1.7044 on perl 5.034000 built for x86_64-linux
  Work directory is /work/x/opt/default/perl5/.cpanm/work/1633573779.21238
  You have make /usr/bin/make
  You have /usr/bin/wget
  You have /usr/bin/tar: tar (GNU tar) 1.26
  Copyright (C) 2011 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason. You have /usr/bin/unzip Searching Archive::Zip () on cpanmetadb ... --2021-10-06 21:29:39-- http://cpanmetadb.plackperl.org/v1.0/package/Archive::Zip Resolving cpanmetadb.plackperl.org (cpanmetadb.plackperl.org)... 151.101.50.217 Connecting to cpanmetadb.plackperl.org (cpanmetadb.plackperl.org)|151.101.50.217|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 469 [text/yaml] Saving to: ‘STDOUT’

100%[===============================================================================================>] 469 --.-K/s in 0s

2021-10-06 21:29:39 (95.7 MB/s) - written to stdout [469/469]

--> Working on Archive::Zip Fetching https://www.cpan.org/authors/id/P/PH/PHRED/Archive-Zip-1.68.tar.gz ... --2021-10-06 21:29:39-- https://www.cpan.org/authors/id/P/PH/PHRED/Archive-Zip-1.68.tar.gz Resolving www.cpan.org (www.cpan.org)... 151.101.50.132, 2a04:4e42:c::644 Connecting to www.cpan.org (www.cpan.org)|151.101.50.132|:443... connected. ERROR: cannot verify www.cpan.org's certificate, issued by ‘/C=US/O=Let's Encrypt/CN=R3’: Issued certificate has expired. To connect to www.cpan.org insecurely, use `--no-check-certificate'. OK Unpacking Archive-Zip-1.68.tar.gz

gzip: stdin: unexpected end of file /usr/bin/tar: Child returned status 1 /usr/bin/tar: Error is not recoverable: exiting now ! Failed to unpack Archive-Zip-1.68.tar.gz: no directory ! Failed to fetch distribution Archive-Zip-1.68

[skirmess]

CPAN is no longer available over HTTP. As a workaround we can completely turn off the certificate check with the .wgetrc

$ cat ~.wgetrc
check_certificate = off

[miyagawa]

setting PERL_LWP_SSL_VERIFY_HOSTNAME to 0 will disable the SSL hostname verification. https://metacpan.org/pod/LWP#PERL_LWP_SSL_VERIFY_HOSTNAME

CPAN is still available over HTTP. The index pages redirect to HTTPS, but individual files for modules and indexes are available over HTTP.

Try: curl -svo /dev/null http://www.cpan.org/modules/02packages.details.txt.gz and you will see 200 OK, not 301 redirect.

[skirmess]

curl seems to ignore HSTS, wget does not. And, at least for me, cpanm prefers wget over curl

$ wget http://www.cpan.org/modules/02packages.details.txt.gz
URL transformed to HTTPS due to an HSTS policy
--2021-11-27 16:50:38--  https://www.cpan.org/modules/02packages.details.txt.gz

[miyagawa]

Ah that's interesting. I guess you fetched the index page with the wget once and that makes HSTS sticky. There's a --no-hsts option but I'm not sure if it's a good idea to pass that, because if you have the HSTS enabled it means you're likely able to fetch the CPAN index via HTTPS.