GitHub action which tests doublep/eldev on Windows Server 2019 servers has suddenly started to fail connecting to MELPA

[ikappaki]

Some Eldev tests download packages from MELPA (e.g. buttercup), but they suddenly started failing to connect to the repostiory. This was only became obvious when an unrelated commit was made to the codebase after a few months of inactivity.

Example error indicates a connection issue to melpa:

[03:40.937]  Ran Eldev as `d:/a/eldev/eldev/bin/eldev --setup '(eldev-require-version "999.9")' upgrade-self --dry-run' in directory `d:/a/eldev/eldev/test/project-a/'
[03:40.937]  Stdout contents:

[03:40.937]  Stderr contents:
             Process stable.melpa.org not running
             When updating contents of package archive `melpa-stable'
             Failed setup step: evaluating form `(eldev-require-version "999.9")' specified on the command line
             Project `project-a' requires Eldev version 999.9 or newer (this is version 0.9.4snapshot)

[03:40.937]  Process exit code: 1

Preliminary analysis has shown that this is likely to the Let's encrypt certificate used by https://stable.melpa.org whose issuer has expired on the 30th of Sep. There appears to be another certificate that replaces it, but both are visible as described here in this comment.

It appears the Emacs version we install on the GitHub windows 2019 server comes with gnutsl-3.6.12, and can't handle the situation very well:

(output captured by reading the *Messages* buffer after (setq gnutls-log-level 3):

gnutls.c: [3] ocsp signer: subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256, activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC', pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="

gnutls.c: [3] ASSERT: ../../../gnutls-3.6.12/lib/x509/ocsp.c[gnutls_ocsp_resp_get_single]:1650

gnutls.c: [3] ASSERT: ../../gnutls-3.6.12/lib/ocsp-api.c[gnutls_ocsp_status_request_get2]:98

gnutls.c: [3] ASSERT: ../../gnutls-3.6.12/lib/ocsp-api.c[gnutls_ocsp_status_request_get2]:98

gnutls.c: [3] ASSERT: ../../../gnutls-3.6.12/lib/x509/name_constraints.c[gnutls_x509_crt_get_name_constraints]:469

gnutls.c: [3] ASSERT: ../../../gnutls-3.6.12/lib/x509/verify.c[verify_crt]:823

gnutls.c: [3] ASSERT: ../../../gnutls-3.6.12/lib/x509/verify.c[verify_crt]:830

gnutls.c: [3] ASSERT: ../../../gnutls-3.6.12/lib/x509/verify.c[_gnutls_verify_crt_status]:1022

gnutls.c: [1] (Emacs) verification: certificate has expired
gnutls.c: [1] (Emacs) verification: certificate could not be verified
gnutls.c: [1] (Emacs) certificate validation failed: stable.melpa.org

but running it on my windows computer with Emacs compiled with gnutls-3.7.0 does work, correctly trying out the second, un-expired, certificate:

gnutls.c: [3] ocsp signer: subject `CN=R3,O=Let's Encrypt,C=US', issuer `CN=ISRG Root X1,O=Internet
Security Research Group,C=US', serial 0x00912b084acf0c18a753f6d62e25a75f5a, RSA key 2048 bits, signed using RSA-SHA256,
activated `2020-09-04 00:00:00 UTC', expires `2025-09-15 16:00:00 UTC',
pin-sha256="jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0="ESC[0m

gnutls.c: [3] ASSERT: ../../../gnutls-3.7.0/lib/x509/ocsp.c[gnutls_ocsp_resp_get_single]

gnutls.c: [3] ASSERT: ../../gnutls-3.7.0/lib/ocsp-api.c[gnutls_ocsp_status_request_get2]

gnutls.c: [3] ASSERT: ../../gnutls-3.7.0/lib/ocsp-api.c[gnutls_ocsp_status_request_get2]

gnutls.c: [3] ASSERT:
../../../gnutls-3.7.0/lib/x509/name_constraints.c[gnutls_x509_crt_get_name_constraints]

gnutls.c: [3] ASSERT: ../../../gnutls-3.7.0/lib/x509/verify.c[verify_crt]

gnutls.c: [3] ASSERT: ../../../gnutls-3.7.0/lib/x509/verify.c[verify_crt]

gnutls.c: [3] ASSERT: ../../../gnutls-3.7.0/lib/x509/verify.c[_gnutls_verify_crt_status]

gnutls.c: [2] issuer in verification was not found or insecure; trying against trust list

I suspect it is this commit in gnutls-3.7.0 that fixes it https://github.com/gnutls/gnutls/commit/e476e4b7e0de11820123da06ca9fcd0cbb94b8ff

This may also be an issue on non-windows system, though it all depends which of the two issuer certificates is referenced first (on MS-Windows it appears to be the expired one).

Assuming above analysis is correct, a possible solution would be to install an Emacs version on the GitHub windows server bundled with a more recent version of the gnutls library. Perhaps this can be done by switching to msys2 installs. The next Emacs release will not have this issue since it will be bundled with a more recent gnutls version.

[ikappaki]

Confirmed, the issue has been already reported to Emacs devel with bug#51038.

[doublep]

I've disabled testing on Windows for now.

By the way, wouldn't it be enough to ask MELPA to update their certificates, as a workaround only, of course? Or do I misunderstand scope of the problem?

[ikappaki]

I've disabled testing on Windows for now.

By the way, wouldn't it be enough to ask MELPA to update their certificates, as a workaround only, of course? Or do I misunderstand scope of the problem?

Great thanks!

MELPA's certificate is valid and there is no need for them to procure a new one. The issue is with the last official precomplied version of Gnu Emacs for MS-Windows which is shipping with a GnuTLS version that has a bug#1008 it can't handle the current, otherwise valid, issuer configuration having both a non-expired and expired certificate.

[doublep]

I misunderstood then. Thought that MELPA provided two certificates somehow, one expired and the other valid. (Don't ask, I have little understanding of how they work.) So, we have to wait until Emacs is rebuilt with fixed GnuTLS? Does the problem affect every Windows installation, e.g. also if you use Eldev locally?

[ikappaki]

So, we have to wait until Emacs is rebuilt with fixed GnuTLS? Does the problem affect every Windows installation, e.g. also if you use Eldev locally?

There are a few ways to install Emacs locally; one is to download and extract the zip file with the precompiled binaries from the gnu ftp site (this is the version that ships with the GnuTLS version which is susceptible to the issue). The other is to use a package manager for MS-Windows (such as Chocolatey or Scoop) to install Emacs, and hope that they have build the package from source with a recent GnuTLS version. And yet another method is to use the MSYS2 tools to build/install/upgrade packages. It uses pacman for managing packages, so the dependencies are regularly updated. This is what I am using.

Eldev uses the https://github.com/jcs090218/setup-emacs-windows github action to install Emacs on the CI Windows Server, which uses the precomplied binaries from the gnu ftp site mentioned earlier, and thus the issue. We can possibly to Chocolatey/MSYS2 for installing the packages but that I think might mess up the Eldev's test.yml file, and I'm trying to resist so far.

An alternative workaround I've just tested to work on GitHub is to instruct Emacs Network Security Manager (NSM) to accept as valid the certificates for the sites the test suite is trying to contact, but can't think of a suitable location to place them other than at the top of the eldev.el file itself, which I don't find of good taste, especially for a workaround that only affects the github windows CI but nothing else. This can be converted it into a function of course hidden somewhere in Eldev that only takes action if running on windows with that particular Emacs signature ... any suggestions?

(setq nsm-temporary-host-settings
      '((:id "sha1:24c7a1e6369fef1e50de1deb1a9e5bbc68f80c8b" :fingerprints
             ("sha1:cf:5a:90:ac:6f:03:dd:c2:55:9b:d7:bf:0e:f1:bc:73:4b:99:60:ce")
             :host "orgmode.org:443" :conditions
             (:expired :invalid :verify-cert))
        (:id "sha1:ccfd2cc2a08c89a9b10969be8c1e926954d53e28" :fingerprints
             ("sha1:0c:bd:68:cb:c0:01:e2:ad:93:0d:b9:3b:77:09:2e:47:9c:de:6b:28")
             :host "stable.melpa.org:443" :conditions
             (:expired :invalid :verify-cert))
        (:id "sha1:6d4eb958390599243ba9f5035cb671fa8dd6a93a" :fingerprints
             ("sha1:56:41:11:79:62:b9:85:66:f8:9e:e4:3b:39:2d:5f:a6:a5:c7:e9:2d")
             :host "elpa.gnu.org:443" :conditions
             (:expired :invalid :verify-cert))))

[jrootabega]

What about in the ~/.emacs.d/network-security.data file, or wherever the CI system has nsm-settings-file configured?

[doublep]

I usually don't hold high hopes for Emacs upstream, but wouldn't they release recompiled binaries soon? This sounds like a critical bug to me, and as I understand all they need to do is to repack with already fixed GnuTLS. Or am I wrong?

[ikappaki]

Hi @jrootabega

What about in the ~/.emacs.d/network-security.data file, or wherever the CI system has nsm-settings-file configured?

That can be another option, but due to the power of Eldev to isolate environments dynamically, there is no single network-security.data path to consider, so we will need a little bit of thinking how to achieve this without polluting the current setup.

Hi @doublep,

I usually don't hold high hopes for Emacs upstream, but wouldn't they release recompiled binaries soon? This sounds like a critical bug to me, and as I understand all they need to do is to repack with already fixed GnuTLS. Or am I wrong?

As the conversion has evolved in Emacs bug#51038 over the last couple of days, there won't be any updates to the existing 27.2 MS-Windows binaries. It turned out these are not considered to be official releases, and as thus unsupported.

I've raised #56 to switch to msys2 until a new version becomes available, I found this to be the path with the least intervention, if you could please have a look ...

[doublep]

@ikappaki: thank you, I merged the PR.