Certificate error when trying to install package

[Monokai]

Since a few days I'm getting reports of people who cannot install my package Theme - Monokai Pro anymore. This is the console output:

Package Control: Error downloading package. HTTP exception InvalidCertificateException (Host packages.monokai.pro returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548))) downloading https://packages.monokai.pro/Theme%20-%20Monokai%20Pro/1.1.19/Theme%20-%20Monokai%20Pro.sublime-package.
error: Package Control

Unable to download Theme - Monokai Pro. Please view the console for more details.

The site https://monokai.pro and subdomain https://packages.monokai.pro are encrypted by Let's Encrypt. I have not changed the server lately.

Any ideas?

[deathaxe]

Some LetsEncrypt root certificates expired recently. Seems your site is effected. It was a quite reasonable topic in tech media.

[Monokai]

I see now. I've read the Let's Encrypt update, but I'm not sure whether I should do something or the package control program? Let's Encrypt says the transition should work without any problems.

[deathaxe]

Package Control uses root certificates from client's operating system via oscrypto library or by directly using the OS's API to download packages. I can install Monokai Pro without issues on Windows for instance.

On some systems Package Control caches root certificates extracted from OS's key chain for 7 days.

Maybe those not yet expired cache files still contain only outdated Lets Encrypt certificates? I would expect the new ones to have been published earlier though.

Those users could try to manually remove the cache file(s).

They should be located in ST's cache directory (Data/Cache/Package Control/).

[rwols]

On ubuntu you can run this:

https://superuser.com/a/1679205

Unfortunately, it's the responsibility of the client to fix it on their end.

[Monokai]

Thank you. Tried a lot, but I've still got errors on macOS while installing the theme, despite having cleared the cache (in ~/Library/Caches/Sublime\ Text\ 3/Cache/Package\ Control). Weird because the site itself uses the same certificate and the theme is reachable via https://packagecontrol.io/packages/Theme%20-%20Monokai%20Pro. Any ideas how the force clear the cache in macOS?

[deathaxe]

Older revisions of PC used to store cached certificates in the User package. Maybe some of those still exists and being used?

[Monokai]

Thanks so far. Still no dice, I also force renewed the certificate but it didn't solve anything.

I found a test site that checks whether you can connect or not: https://expired-r3-test.scotthelme.co.uk and indeed it does not work in Safari on my computer (2018 macOS 11.6). It does on Firefox however (apparently because it uses its own keystore).

So if I understand correctly this leaves me with a certificate that's broken on my computer and clients' computers without me being able to fix it on the servers' end. Any other ideas for a possible fix would be greatly appreciated.

[rwols]

The fix is to ask clients to update their certificate store. Older devices that don’t receive updates are doomed.

[rchl]

A little drastic but you could switch to an alternative certificate (provider) that doesn't have this issue.

[FichteFoll]

Users could also try specifying a different downloader precedence in PC's settings to find a downloader that is not affected, e.g. curl.

If "urllib" refers to the Python-internal urllib, then it will result in the OpenSSL version that ST's Python was build with. For the latest ST4 dev build, that is 1.1.1 in the Python 3.8 and 1.0.2 in the Python 3.3 plugin hosts, where the latter is affected by this problem. https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

Edit: Seems to work for me, though.

Python 3.3>>> from urllib.request import urlopen; urlopen("https://packages.monokai.pro/Theme%20-%20Monokai%20Pro/1.1.19/Theme%20-%20Monokai%20Pro.sublime-package").status
200

Edit2: I forgot I need to specify some CAs.

Python 3.3>>> from urllib.request import urlopen; urlopen("https://packages.monokai.pro/Theme%20-%20Monokai%20Pro/1.1.19/Theme%20-%20Monokai%20Pro.sublime-package", cafile="/usr/share/ca-certificates/trust-source/mozilla.trust.p11-kit").status
[…]
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:548)>

I created a core issue to track this, btw. https://github.com/sublimehq/sublime_text/issues/4903

[FichteFoll]

I should probably mention that performing the first workaround mentioned in the blog post, namely removing the problematic root CS from your OS's trust strore, is also an option.

Alternatively, PC could release an update that blacklists this root CA in particular before passing the system's trust store to the downloader, but I'd leave that decision to wbond since he has way more expertise regarding potential consequences. The advantage of that would be that users don't need to take action, even if they remain on an older ST build (assuming the core issue is addressed soon-ish). But getting a non-LE cert would be easier and quicker.

[Monokai]

Thanks @FichteFoll for digging deeper and for creating the core issue. It does make sense that my system otherwise works fine, but only has issues with Package Control (via the affected OpenSSL 1.0.2).

I think asking users to remove the problematic root CS from the trust store is suboptimal, as it requires some technical actions from the user. I could install a new certificate, but I really like the self-renewable Let's Encrypt's setup. Curious what @wbond thinks.

[FichteFoll]

I mentioned my findings on Discord. Here's what he said:

[16:25] FichteFoll: PC could also push an update that skips this problematic root when importing from the OS trust store. that would be the most immediate no-user-action-required fix
[16:38] wbond: That a lot of extra work and testing for me, and my limited time is mostly around PC4 at this point. So I understand the scope - this only affects a couple of packages, correct?
[16:39] wbond: It would seem the simplest option would be to have those package authors pay $10 a get a non-LE cert
[16:39] wbond: Rather than me spend $500-800 of labor of working around it for them
[16:42] wbond: Buying a non-LE cert could have the issue results by the EOD pretty easily. Rolling out a new release would probably take a week minimum, due to personal and work commitments I have.
[16:49] FichteFoll: I agree

[Monokai]

OK, clear. That puts the onus on me again. Will try to reconfigure my server with a different certificate over the weekend.

[wbond]

From a practical standpoint, your packages can be hosted anywhere with a secure certificate.

I have no problem with Package Control being modified to work around the LE issue here, but it will take much longer than replacing a cert or two. Testing and cutting a new release of Package Control is a rather non-trivial process and will likely take a few weeks. If anything is broken along the way, Package Control won’t be able to update itself and all installs will be in an orphaned state until each user installs fresh.

Thus from a pragmatic standpoint switching cert providers feels like the best approach. It costs $10 or so, and should take about 1-2 hours (at most) of work for one web server. Alternatively you can keep LE and wait for a PC release, although I can’t give a firm time when that would happen. It may be a few weeks, as right now I’ve got many things going on in my personal life.

[Monokai]

Thank you, that's correct and I totally understand. Glad I now know what the problem is. Will update on my end! :)

[Monokai]

So I've updated it 2 weeks ago with a SSL from Buypass, which doesn't seem to give any problems. Thanks for your help all!