Closed dhalbert closed 11 months ago
@dhalbert - on https://github.com/adafruit/circuitpython/issues/8299, the internet test was performed using a MatrixPortal S3. That board doesn't run nina-fw. Is this issue being within nina-fw relevant?
Looking at https://github.com/adafruit/nina-fw/pull/51, I see that it replaces an existing certificate. When I enter the existing certificate, it does not show up as expired, but instead shows up as expiring in 2031. Was the certificate revoked?
I believe it was revoked - @martymcguire and @jwcooper would know for sure.
I believe Digicert updated their root and intermediate CA certificates in 2023, so we picked those up in this refresh: https://knowledge.digicert.com/generalinformation/digicert-root-and-intermediate-ca-certificate-updates-2023.html
@dhalbert - on adafruit/circuitpython#8299, the internet test was performed using a MatrixPortal S3. That board doesn't run nina-fw. Is this issue being within nina-fw relevant?
CircuitPython uses NINA-FW as a submodule to pick up the latest roots.pem
for all our boards including native wifi like S3. So, yep, changes here affect CircuitPython certs.
I wondered if there was a minor change or not, sadly my diff software couldn't pull off the moved block detection well enough, but this is the latest roots.pem from android, taken according to the instructions at https://github.com/arduino/nina-fw/blob/master/README.md#build-a-new-certificate-list-based-on-the-google-android-root-ca-list PR Added #56, happy to cancel it, but thought easiest to just pull fresh certs. Using the instructions to test the certs I verified api.github.com works on my branch, and not on master, io.adafruit.com works on both.
As a bit of a personal challenge I wanted to see if I could get WinMerge etc to do moved block detection well enough to understand the roots bundle changes. Yes and no.
I decided to play with Noteable (Jupyter notebooks, but works with chatgpt via plugin), and attempt to do moved block detection and looking up old certs, it worked well. I've also added the older certs to see what happened compared to the past.
This made me realise there was not a small change and we should probably accept the newer file rather than patching individual certs. I'd love to know what the unknown certs are (all removed by my update) so if anyone wants to suggest extensions to the notebook please do. https://app.noteable.io/f/dcd52f2d-0761-4287-b90c-ac419f130c16/root-certs.ipynb
Number of moved certificates: 23
{'USERTrust ECC Certification Authority': 'USERTrust ECC Certification Authority', 'GlobalSign Root CA - R3': 'GlobalSign Root CA - R3', 'Starfield Class 2 CA': 'Starfield Class 2 CA', 'GlobalSign ECC Root CA - R5': 'GlobalSign ECC Root CA - R5', 'USERTrust RSA Certification Authority': 'USERTrust RSA Certification Authority', 'Baltimore CyberTrust Root': 'Baltimore CyberTrust Root', 'COMODO Certification Authority': 'COMODO Certification Authority', 'ISRG Root X1': 'ISRG Root X1', 'DigiCert Assured ID Root CA': 'DigiCert Assured ID Root CA', 'Comodo AAA Services root': 'Comodo AAA Services root', 'Amazon Root CA 3': 'Amazon Root CA 3', 'DigiCert High Assurance EV Root CA': 'DigiCert High Assurance EV Root CA', 'Starfield Root Certificate Authority - G2': 'Starfield Root Certificate Authority - G2', 'COMODO ECC Certification Authority': 'COMODO ECC Certification Authority', 'DigiCert Assured ID Root G2': 'DigiCert Assured ID Root G2', 'GlobalSign Root CA - R6': 'GlobalSign Root CA - R6', 'DigiCert Trusted Root G4': 'DigiCert Trusted Root G4', 'GlobalSign Root CA': 'GlobalSign Root CA', 'DigiCert Assured ID Root G3': 'DigiCert Assured ID Root G3', 'DigiCert Global Root G2': 'DigiCert Global Root G2', 'COMODO RSA Certification Authority': 'COMODO RSA Certification Authority', 'DigiCert Global Root G3': 'DigiCert Global Root G3', 'Amazon Root CA 1': 'Amazon Root CA 1'}
Number of new certificates: 12
dict_keys(['GlobalSign ECC Root CA - R4', 'Starfield Services Root Certificate Authority - G2', 'GlobalSign Root R46', 'DigiCert Global Root CA', 'GTS Root R4', 'GTS Root R3', 'ISRG Root X2', 'GTS Root R2', 'Amazon Root CA 4', 'GlobalSign Root E46', 'GTS Root R1', 'Amazon Root CA 2'])
Number of removed certificates: 25
dict_keys(['Unknown1', 'Unknown2', 'Entrust Root Certification Authority', 'Unknown3', 'Unknown4', 'Unknown5', 'GlobalSign Root CA - R2', 'Go Daddy Root Certificate Authority - G2', 'Unknown6', 'Cybertrust Global Root', 'Unknown7', 'Entrust Root Certification Authority - EC1', 'Entrust Root Certification Authority - G2', 'Entrust.net Premium 2048 Secure Server CA', 'AffirmTrust Commercial', 'AffirmTrust Networking', 'AffirmTrust Premium', 'AffirmTrust Premium ECC', 'Unknown8', 'Go Daddy Class 2 CA', 'GlobalSign ECC Root CA - R4', 'GTS Root R1', 'GTS Root R2', 'GTS Root R3', 'GTS Root R4'])
Having same issue connecting SSL to a website behind a cloudflare certificate
When running through this tutorial https://learn.adafruit.com/adafruit-pyportal/internet-connect#connect-to-wifi-17-4 I get a runtime error:
adafruit_requests.py line 515 in _get_socket.
RuntimeError: sending request failed
Running nina 1.7.5 Circuit Python 8.26 on PyPortal
It worked fine before the URL in question switched to use cloudflare
@crhuber Could you try this NINA-FW firmware update? It has an updated certificate list, and works with the github URL https://api.github.com/repos/adafruit/circuitpython. That returns a lot of JSON so I had to trim the rest of the test program to avoid memory issues.
Unzip this and update the ESP32 AirLift firmware: NINA_W102-1.7.7.bin.zip
If it works I will submit a PR to nina-fw.
I tested with 1.7.7 Nina FW and Circuit Python 8.2.6 it works with this url https://api.github.com/repos/adafruit/circuitpython.
but fails for a different url with a lets encrypt cert provided by cloudflare
Could you please give the failing URL, or at least a redacted version so we can try talking to the same server and looking at its certificate chain?
@dhalbert try this one https://api.intra.io/api/v1/healthz
@crhuber I see that problem too, on a Metro M4 AirLift Lite using the NINA-FW firmware I posted above. But there is something strange going on with DNS for
The site works with api.intra.io
:curl
:
$ curl https://api.intra.io/api/v1/healthz
{"status":"ok"}
But it doesn't work with openssl s_client
[EDIT: fixed command] [EDIT2: command should only refer to the server, not the whole URL: see https://github.com/adafruit/nina-fw/issues/55#issuecomment-1728349451]
$ openssl s_client -showcerts -connect api.intra.io/api/v1/healthz:443 < /dev/null #EDIT2: arg is WRONG
40877E77527F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/bio_addr.c:738:Name or service not known
connect:errno=22
I also tried that URL on a Metro ESP32-S2, and it works. That board is using CircuitPython with a roots.pem
that is identical to the roots.pem
used by the test 1.7.7 NINA-FW.
The cert chain as reported by Firefox is:
*.intra.io
server cert <- Let's Encrypt E1 <- ISRG Root X2 (using elliptic keys)
The ISRG Root X2 cert is in our roots.pem
. But since openssl s_client
cannot even look up the host, and it does work with CircuitPython ESP32-S2, I don't think it's a cert chain issue.
Do you have any idea of why openssl s_client
can't look up the host? I'm hypothesizing the same issue is affecting NINA-FW.
@dhalbert same with my domains, maybe easier to diagnose/test. good-enough.cloud is cloudflare letsencrypt E1, but good-enough.technology is certbot letsencrypt R3. Both should resolve okay.
-Edit-
Certbot version:
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = good-enough.technology
verify return:1
---
cloudflare version:
CONNECTED(00000003)
depth=3 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X2
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = E1
verify return:1
depth=0 CN = good-enough.cloud
verify return:1
---
@dhalbert I think the command you used for openssl was incorrect. I dont think you need the url path in the command.
Also on cloudflare I tried disabling TLS 1.3 and changing Minimum TLS to 1.2 (from 1.0) but no difference
@crhuber you are right; I was cutting and pasting the commands and missed that. Will investigate further.
If I set esp._debug = True
in the standard Internet test (which I slightly modified to fetch only the intra.io URL), it appears that the NINA-FW code is crashing:
Fetching and parsing json from https://api.intra.io/api/v1/healthz
*** Get host by name
*** Get socket
Allocated socket #0
*** Socket connect mode 2
*** Open socket to api.intra.io 443 2
*** Closing socket #0
Traceback (most recent call last):
File "adafruit_requests.py", line 515, in _get_socket
File "adafruit_requests.py", line 748, in connect
File "adafruit_esp32spi/adafruit_esp32spi_socket.py", line 77, in connect
File "adafruit_esp32spi/adafruit_esp32spi.py", line 819, in socket_connect
File "adafruit_esp32spi/adafruit_esp32spi.py", line 715, in socket_open
File "adafruit_esp32spi/adafruit_esp32spi.py", line 341, in _send_command_get_response
File "adafruit_esp32spi/adafruit_esp32spi.py", line 297, in _wait_response_cmd
File "adafruit_esp32spi/adafruit_esp32spi.py", line 206, in _wait_for_ready
TimeoutError: ESP32 not responding
I am going to split off the https://api.intra.io/api/v1/healthz problem into a new issue, because the original github issue is solvable. But we do need to debug the intra.io issue.
After the certificate replacement in #51, the basic "Internet Test" used across all our CircuitPython wifi boards no longer connects to GitHub. See https://github.com/adafruit/circuitpython/issues/8299.
Looking at #51, I see that it replaces an existing certificate. When I enter the existing certificate, it does not show up as expired, but instead shows up as expiring in 2031.
Was the certificate revoked?
@brentru How did you decide to replace that particular certificate? I am wondering if a still-legit cert was replaced by accident.