GitLab CI pipeline broken (due to HTTPS by default?)

[dboehmer]

The GitLab CI pipeline of my project Coocook is now broken. After a hint by @FErki it looks like this is because this project adopting HTTPS by default (#164 #167).

The CI log: https://gitlab.com/coocook/coocook/-/jobs/7770418332

The bit with the error:

$ PERL5LIB=perl5/lib/perl5/ cpanm --notest --local-lib perl5/ local::lib $( [ -n "$INSTALL_DEVEL_COVER" ] && echo Devel::Cover )
! Finding local::lib on cpanmetadb failed.
LWP will support https URLs if the LWP::Protocol::https module
is installed.

I tried to reproduce this in Docker locally but cpanm works like a charm:

$ sudo docker run -it perl:5.40 bash
root@c18b3d0cb6dc:/usr/src/app# PERL5LIB=perl5/lib/perl5/ cpanm --notest --local-lib perl5/ local::lib
--> Working on local::lib
Fetching https://www.cpan.org/authors/id/H/HA/HAARG/local-lib-2.000029.tar.gz ... OK
Configuring local-lib-2.000029 ... OK
Building local-lib-2.000029 ... OK
Successfully installed local-lib-2.000029
1 distribution installed

Notice that in GitLab CI I keep the CPAN dependencies in a cache. I suspect that some dependency causes the problem in CI because it is present or it is outdated. I might possibly fix the issue by clearing the cache but I want to fix the root cause and therefor don’t dare to touch the cache.

What exactly might be causing the issue? How to reproduce this locally? How to reliably fix this?

[dboehmer]

I read the comment by user Ferki again and could now successfully reproduce the problem: This issue is when the LWP dist is already installed but LWP::Protocol::https is not.

$ sudo docker run -it perl:5.40 bash
[sudo] password for daniel: 
root@34eb7d01f637:/usr/src/app# cpanm LWP::UserAgent
--> Working on LWP::UserAgent
Fetching https://www.cpan.org/authors/id/O/OA/OALDERS/libwww-perl-6.77.tar.gz ... OK
Configuring libwww-perl-6.77 ... OK
==> Found dependencies: URI, HTTP::Response, HTTP::Cookies, LWP::MediaTypes, Encode::Locale, Try::Tiny, HTTP::Status, WWW::RobotRules, Test::RequiresInternet, HTML::Entities, HTTP::Request::Common, Test::Needs, HTTP::Request, HTTP::Daemon, Test::Fatal, HTTP::CookieJar::LWP, File::Listing, HTTP::Date, HTTP::Negotiate, HTML::HeadParser, URI::Escape, Net::HTTP
...
Building and testing libwww-perl-6.77 ... OK
Successfully installed libwww-perl-6.77
31 distributions installed
root@34eb7d01f637:/usr/src/app# cpanm local::lib
! Finding local::lib on cpanmetadb failed.
LWP will support https URLs if the LWP::Protocol::https module
is installed.

Seems like a workaround could be to install LWP::Protocol::https with LWP in the first place. Both missing is fine, both installed is fine as well. I’d prefer not to handle this in my project though but have docker-perl fix this.

Shouldn’t the patched cpanm detect that LWP::Protocol::https is missing and use one of the working fallbacks for LWP missing or fallback to HTTP (not because of transport issues what would be insecure but because of a locally missing dependency)?

[zakame]

Hi @dboehmer, thanks for the report!

Yep, I think this is an oversight of the HTTPS patch - looking at the cpanm doc, we can plainly see why this happens:

Normally with --lwp, --wget and --curl options set to true (which is the default) cpanm tries LWP, Wget, cURL and HTTP::Tiny (in that order) and uses the first one available.

Easiest solution for this is to turn off the patched Menlo::CLI::Compat try_lwp flag to off by default so it falls back to external wget/curl only - let me write a test and PR for this 🙇

[zakame]

For the meantime though, could you try calling cpanm calls in https://gitlab.com/coocook/coocook/-/blob/master/.gitlab-ci.yml?ref_type=heads#L61 and elsewhere with --no-lwp for now?

[dboehmer]

I could successfully fix my pipeline by setting PERL_CPANM_OPT=--no-lwp in the GitLab CI settings. :tada: I preferred this way over adding commits to my project history to work around this issue.

[zakame]

That's a perfect use of that variable 👍

I'll merge in #170 now and will get this on Docker Hub soon. Thanks again for the reports! 🙇