`https://otau.meethue.com` started using a self-signed certificate

kaechele commented 8 months ago

Signify seems to have started using self-signed certificates for https://otau.meethue.com. Unsure if this is a bug on their end or a new configuration they have adopted going forward.

Either way, currently OTA updates using images hosted on that URL will fail:

debug 2023-12-29 05:55:33: Received MQTT message on 'zigbee2mqtt/bridge/request/device/ota_update/update' with data '{"id":"Bedroom Lamp","transaction":"r82z7-1"}'
info  2023-12-29 05:55:33: Updating 'Bedroom Lamp' to latest firmware
debug 2023-12-29 05:55:33: Received Zigbee message from 'Bedroom Lamp', type 'readResponse', cluster 'genBasic', data '{"dateCode":"20230606","swBuildId":"1.108.5"}' from endpoint 11 with groupID 0
info  2023-12-29 05:55:33: MQTT publish: topic 'zigbee2mqtt/Bedroom Lamp', payload '{"brightness":254,"color":{"h":32,"hue":32,"s":81,"saturation":81,"x":0.4574,"y":0.41},"color_mode":"color_temp","color_temp":366,"color_temp_startup":65535,"last_seen":"2023-12-29T10:55:33.511Z","linkquality":144,"power_on_behavior":"previous","state":"OFF","update":{"installed_version":16786946,"latest_version":16786948,"state":"available"},"update_available":null}'
debug 2023-12-29 05:55:33: Updating to latest '0x0017880103adfb62' (LCT016)
debug 2023-12-29 05:55:33: Using endpoint '11'
debug 2023-12-29 05:55:33: Received Zigbee message from 'Bedroom Lamp', type 'commandQueryNextImageRequest', cluster 'genOta', data '{"fieldControl":0,"fileVersion":16786946,"imageType":268,"manufacturerCode":4107}' from endpoint 11 with groupID 0
info  2023-12-29 05:55:33: MQTT publish: topic 'zigbee2mqtt/Bedroom Lamp', payload '{"brightness":254,"color":{"h":32,"hue":32,"s":81,"saturation":81,"x":0.4574,"y":0.41},"color_mode":"color_temp","color_temp":366,"color_temp_startup":65535,"last_seen":"2023-12-29T10:55:33.739Z","linkquality":144,"power_on_behavior":"previous","state":"OFF","update":{"installed_version":16786946,"latest_version":16786948,"state":"available"},"update_available":null}'
debug 2023-12-29 05:55:33: Got OTA request '{"fieldControl":0,"manufacturerCode":4107,"imageType":268,"fileVersion":16786946}'
debug 2023-12-29 05:55:33: ZigbeeOTA: downloaded main index
debug 2023-12-29 05:55:33: getNewImage for '0x0017880103adfb62', meta {"fileVersion":16786948,"fileSize":267324,"url":"https://otau.meethue.com/storage/ZGB_100B_010C/3a5dab1a-faf6-47a8-9c9a-34f34eec029f/100B-010C-01002604-ConfLight-Lamps_0012.zigbee","sha512":"e9371e1599b193730f5358cdc64671e93042de1ddf669a970c4e9ce76363bea0b7198838fa4584271b3f1586e4d92565a2cc28a90143889576bab6576ef0e2ce"}
debug 2023-12-29 05:55:33: ZigbeeOTA: downloading firmware image from https://otau.meethue.com/storage/ZGB_100B_010C/3a5dab1a-faf6-47a8-9c9a-34f34eec029f/100B-010C-01002604-ConfLight-Lamps_0012.zigbee
debug 2023-12-29 05:55:34: Update of 'Bedroom Lamp' failed (Error: self signed certificate in certificate chain)
info  2023-12-29 05:55:34: MQTT publish: topic 'zigbee2mqtt/Bedroom Lamp', payload '{"brightness":254,"color":{"h":32,"hue":32,"s":81,"saturation":81,"x":0.4574,"y":0.41},"color_mode":"color_temp","color_temp":366,"color_temp_startup":65535,"last_seen":"2023-12-29T10:55:33.739Z","linkquality":144,"power_on_behavior":"previous","state":"OFF","update":{"installed_version":16786946,"latest_version":16786948,"state":"available"},"update_available":null}'
info  2023-12-29 05:55:34: MQTT publish: topic 'zigbee2mqtt/bridge/response/device/ota_update/update', payload '{"data":{"id":"Bedroom Lamp"},"error":"Update of 'Bedroom Lamp' failed (self signed certificate in certificate chain)","status":"error","transaction":"r82z7-1"}'
error 2023-12-29 05:55:34: Update of 'Bedroom Lamp' failed (self signed certificate in certificate chain)
debug 2023-12-29 05:55:34: Error: self signed certificate in certificate chain
    at Function.AxiosError.from (/app/node_modules/axios/lib/core/AxiosError.js:89:14)
    at RedirectableRequest.handleRequestError (/app/node_modules/axios/lib/adapters/http.js:606:25)
    at RedirectableRequest.emit (node:events:513:28)
    at ClientRequest.eventHandlers.<computed> (/app/node_modules/follow-redirects/index.js:14:24)
    at ClientRequest.emit (node:events:513:28)
    at TLSSocket.socketErrorListener (node:_http_client:494:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Certificate Chain

``` ❯ openssl s_client -showcerts -connect otau.meethue.com:443 CONNECTED(00000003) depth=2 C = NL, O = Philips Hue, CN = root verify error:num=19:self-signed certificate in certificate chain verify return:1 depth=2 C = NL, O = Philips Hue, CN = root verify return:1 depth=1 C = NL, O = Philips Hue, CN = intermediate verify return:1 depth=0 C = NL, O = Philips Hue, CN = otau.meethue.com verify return:1 --- Certificate chain 0 s:C = NL, O = Philips Hue, CN = otau.meethue.com i:C = NL, O = Philips Hue, CN = intermediate a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Dec 28 08:19:26 2023 GMT; NotAfter: Dec 27 08:19:26 2024 GMT -----BEGIN CERTIFICATE----- MIICOzCCAeGgAwIBAgICEIYwCgYIKoZIzj0EAwIwOjELMAkGA1UEBhMCTkwxFDAS BgNVBAoMC1BoaWxpcHMgSHVlMRUwEwYDVQQDDAxpbnRlcm1lZGlhdGUwHhcNMjMx MjI4MDgxOTI2WhcNMjQxMjI3MDgxOTI2WjA+MQswCQYDVQQGEwJOTDEUMBIGA1UE ChMLUGhpbGlwcyBIdWUxGTAXBgNVBAMTEG90YXUubWVldGh1ZS5jb20wWTATBgcq hkjOPQIBBggqhkjOPQMBBwNCAASsjJWztRR+y0Fh1moAEFInY7Hm0Bs2sNVlWwbh G1ND0xM5q88rRDPU/0uhseC1Q8zrIG5odks/qKFeyWPuluvmo4HSMIHPMAwGA1Ud EwEB/wQCMAAwHQYDVR0OBBYEFODTXkF2YAGuNZdIYa9CXgVRHYg5MFsGA1UdIwRU MFKAFE5wTAXeCEvl+iOFRD6V/DHv8j+MoTakNDAyMQswCQYDVQQGEwJOTDEUMBIG A1UECgwLUGhpbGlwcyBIdWUxDTALBgNVBAMMBHJvb3SCAhAEMA4GA1UdDwEB/wQE AwIFoDAWBgNVHSUBAf8EDDAKBggrBgEFBQcDATAbBgNVHREEFDASghBvdGF1Lm1l ZXRodWUuY29tMAoGCCqGSM49BAMCA0gAMEUCICbyQzlJ3qB+hXQpK1Div1LPrCA3 4cBeGDph2y2gvJhYAiEA7R2WE6/qVsxlkL5h4Puh+yxp/ofs0laGMzy9FzQczYM= -----END CERTIFICATE----- 1 s:C = NL, O = Philips Hue, CN = intermediate i:C = NL, O = Philips Hue, CN = root a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Sep 21 09:35:35 2021 GMT; NotAfter: Nov 10 09:35:35 2026 GMT -----BEGIN CERTIFICATE----- MIIBwTCCAWigAwIBAgICEAQwCgYIKoZIzj0EAwIwMjELMAkGA1UEBhMCTkwxFDAS BgNVBAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MB4XDTIxMDkyMTA5MzUz NVoXDTI2MTExMDA5MzUzNVowOjELMAkGA1UEBhMCTkwxFDASBgNVBAoMC1BoaWxp cHMgSHVlMRUwEwYDVQQDDAxpbnRlcm1lZGlhdGUwWTATBgcqhkjOPQIBBggqhkjO PQMBBwNCAATApYP6FMJcZkAMz98aJIJ08gpXW3FT47ikdQ4KlmztyYeTyKPWf/sP wiTeAu5nr2y8Gj0jYhk/6FNenuRdV+w0o2YwZDAdBgNVHQ4EFgQUTnBMBd4IS+X6 I4VEPpX8Me/yP4wwHwYDVR0jBBgwFoAUCWQVgAJXOKupOv75xaJrwAof984wEgYD VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwCgYIKoZIzj0EAwIDRwAw RAIgMSvQr5cUypLgd752OzS8WF0r6duHV9el8R1YDHl2qJUCIHsvs0CiH6Xg/+9P k1lIs2YkrWB4faKk7/rEGnv4RAx1 -----END CERTIFICATE----- 2 s:C = NL, O = Philips Hue, CN = root i:C = NL, O = Philips Hue, CN = root a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Aug 25 07:59:43 2016 GMT; NotAfter: Jan 5 07:59:43 2068 GMT -----BEGIN CERTIFICATE----- MIIBwDCCAWagAwIBAgIJAJtrMkoTxs+WMAoGCCqGSM49BAMCMDIxCzAJBgNVBAYT Ak5MMRQwEgYDVQQKDAtQaGlsaXBzIEh1ZTENMAsGA1UEAwwEcm9vdDAgFw0xNjA4 MjUwNzU5NDNaGA8yMDY4MDEwNTA3NTk0M1owMjELMAkGA1UEBhMCTkwxFDASBgNV BAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MFkwEwYHKoZIzj0CAQYIKoZI zj0DAQcDQgAEENC1JOl6BxJrwCb+YK655zlM57VKFSi5OHDsmlCaF/EfTGGgU08/ JUtkCyMlHUUoYBZyzCBKXqRKkrT512evEKNjMGEwHQYDVR0OBBYEFAlkFYACVzir qTr++cWia8AKH/fOMB8GA1UdIwQYMBaAFAlkFYACVzirqTr++cWia8AKH/fOMA8G A1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA0gAMEUC IQDcGfyXaUl5hjr5YE8m2piXhMcDzHTNbO1RvGgz4r9IswIgFTTw/R85KyfIiW+E clwJRVSsq8EApeFREenCkRM0EIk= -----END CERTIFICATE----- --- Server certificate subject=C = NL, O = Philips Hue, CN = otau.meethue.com issuer=C = NL, O = Philips Hue, CN = intermediate --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 1795 bytes and written 400 bytes Verification error: self-signed certificate in certificate chain --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 256 bit This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 19 (self-signed certificate in certificate chain) --- ```

kaechele commented 8 months ago

FWIW, the SHA512 sum still matches. So file integrity seems unaffected.

``` ❯ wget --no-check-certificate https://otau.meethue.com/storage/ZGB_100B_010C/3a5dab1a-faf6-47a8-9c9a-34f34eec029f/100B-010C-01002604-ConfLight-Lamps_0012.zigbee --2023-12-29 11:59:39-- https://otau.meethue.com/storage/ZGB_100B_010C/3a5dab1a-faf6-47a8-9c9a-34f34eec029f/100B-010C-01002604-ConfLight-Lamps_0012.zigbee Resolving otau.meethue.com (otau.meethue.com)... 35.241.40.143 Connecting to otau.meethue.com (otau.meethue.com)|35.241.40.143|:443... connected. WARNING: The certificate of ‘otau.meethue.com’ is not trusted. WARNING: The certificate of ‘otau.meethue.com’ doesn't have a known issuer. HTTP request sent, awaiting response... 200 OK Length: 267324 (261K) [application/octet-stream] Saving to: ‘100B-010C-01002604-ConfLight-Lamps_0012.zigbee’ 100B-010C-01002604-ConfLight-Lamps_0012.zigb 100%[============================================================================================>] 261.06K --.-KB/s in 0.1s 2023-12-29 11:59:39 (2.29 MB/s) - ‘100B-010C-01002604-ConfLight-Lamps_0012.zigbee’ saved [267324/267324] ~/Downloads ❯ sha512sum 100B-010C-01002604-ConfLight-Lamps_0012.zigbee e9371e1599b193730f5358cdc64671e93042de1ddf669a970c4e9ce76363bea0b7198838fa4584271b3f1586e4d92565a2cc28a90143889576bab6576ef0e2ce 100B-010C-01002604-ConfLight-Lamps_0012.zigbee ```

kaechele commented 8 months ago

If this is, in fact a persistent change on Signify's end we'd need to implement passing of a custom ca option to the getAxios function in https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/src/lib/ota/common.ts#L554 As well as in https://github.com/Koenkk/zigbee-herdsman-converters/blob/master/src/lib/ota/common.ts#L82

If we need to manage custom CA certs it would probably be sensible to add some metadata to this repository for hosts that require custom certs. That way the certs can be tracked in the Git history of this repo for transparency.

Koenkk commented 8 months ago

I would propose to:

Add link of the certificate to https://github.com/Koenkk/zigbee-OTA/blob/master/index.json
Update zigbee-herdsman-converters to use it

kaechele commented 8 months ago

I started dabbling a bit with it yesterday: https://github.com/kaechele/zigbee-OTA/commit/643f00b2683cf661ecc8d6ac9ca9955b3cb5f4ab

What this basically proposes is that this repository would store all custom CA certs and create a bundle that zigbee-herdsman-converters can use.

This proposal would require the manual effort of pushing any new certificates to this repo and creating the bundle. The fetching of the certificate could be automated as well, but I believe it's not too bad of an idea to do it by hand.

The benefit of this would be an actual increase in security. Manually vetted custom CAs and pinning downloads from the respective hosts would probably implement what Signify is intending with this change: They control the entire signature path for firmware image delivery. Ideally we'd checksum the CA cert bundle in this repo so that it can be easily verified.

Your proposal would certainly work as well. But I can think of a few questions I'd have:

Were you intending to link the CA cert with every entry in index.json that needs it (i.e. every Signify image link)? That, I think, would create a lot of redundancy in the index.json
Where would you link to for the CA certs? Would you want to store the CA cert in this repo? As far as I know there is no downloadable URL for the Signify CA cert we could hotlink to.

I think a combined approach could be feasible as well. Mirror the CA certs in this repo for transparency and change tracking, link to the raw files on GitHub for zigbee-herdsman-converters to use in the fashion that you suggested (link in index.json).

Koenkk commented 8 months ago

What about adding a certs.json to this repo which will be checked before downloading an OTA? Something like this:

{
  "host": "otau.meethue.com",
  "ca": "...."
}

kaechele commented 8 months ago

I thought of that as well, but wasn't sure if this would be an acceptable approach for you, since it adds a custom approach compared to the simple CA cert bundle approach.

The benefits of doing it with a certs.json is that we could actually pin downloads from those hosts to specific CA certs. The ca option we can provide to the Axios httpsAgent options accepts a PEM encoded string as a CA certificate. We could pull that string directly from the JSON ca field (i.e. storing the cert right there in the JSON file, essentially providing a JSON based CA cert bundle). By only supplying the one CA cert we know this host should be using we can provide some additional security (i.e OTA will fail if a cert from another, unexpected CA is presented and someone would have to manually check to see what's going on and if it is malicious)

The CA cert bundle version is a bit simpler. It just expands the list of CAs we trust during download. But it wouldn't bail if suddenly instead of Signify's CA we'd be presented with a download coming from a host presenting a LetsEncrypt certificate, because LetsEncrypt is a trusted CA as well.

Koenkk commented 8 months ago

Do we need to add certs for all hosts when going for the CA bundle approach?

kaechele commented 8 months ago

Only if they are not already trusted by the system's CA cert bundle. For the purposes of this repository, and with pretty much any stock NodeJS installation, at this point you would only need to add the Signify CA to be in the clear.

If you look at Line 58 of the scripts/add.js diff in my little experiment you can see how I add the default system CA bundle to the array and then simply push my custom CA bundle (only consisting of the Signify CA at the moment) into the array afterwards (line 60). I do this because the docs on the ca option in the NodeJS TLS API say:

Mozilla's CAs are completely replaced when CAs are explicitly specified using this option.

So I first need to add the default CAs back (from tls.rootCertificates) before I can append anything.

So in zigbee-herdsman-converters it could be implemented in the very same way, we'd just need to add a way to first get the additional CA certs we need (i.e. for now, just the Signify CA). And for that it'd probably make sense to host them in this repo.

Koenkk commented 8 months ago

Does this mean we also need to keep updating cacerts.pem?

kaechele commented 8 months ago

Yes, that file would always need to be updated (i.e. regenerated) when we add another CA cert.

Concatenating all CA certs into one single cacert.pem has the benefit of only needing to download one file, even if we need to add multiple CA certs. Basically I was trying to think ahead here.

Ultimately, this could be made part of scripts/add.js to add new CAs. I did look into implementing a Proof of Concept but right now I'm lacking the time while I'm travelling.

Another side note: In this case with Signify we have the Root CA certificate due to the fact that they configured their webservers to actually send out the Root CA within the certificate chain. Technically, this would be considered a misconfiguration, as you are generally supposed to already have and trust the Root CA on your machine. In order for this to make sense with other manufacturers possibly doing the same we would need to collect the Root CAs from the manufacturers.

Also, as my mind tends to get carried away, I was also thinking we could add GitHub actions to regularly (maybe weekly?) run scripts/updateall.js to check for any changes (there actually are a few if you'd run it right now). That way we could also detect changes to required CA certs.

I'm certainly happy to support with implementing any of this. Just let me know where I can help.

Koenkk commented 8 months ago

So 'cacert.pem' currently only has to be updated when the Hue certificate changes or when we add a new certificate? I like the automation part, we can run it daily even. Would you mind making a pr for both this repo and zigbee-herdsman-converters?

kaechele commented 8 months ago

Correct.

I'm happy to provide a PR. I may get around to it tomorrow or the day after.

kaechele commented 8 months ago

Working on it as we speak. I am hitting a few complexities I'm hoping to have worked out soon. Stay tuned :)

fredleger commented 8 months ago

just to add to it are you really sure signify is doing that on purpose ? Looking at the cert showed by the site it looks to me they tried to introduce an intermediate CA and forget to add the root CA cert in the chain, but i can be wrong Anyway the fix is needed and will be efficient in the long term.

kaechele commented 8 months ago

just to add to it are you really sure signify is doing that on purpose ?

Can't be entirely sure. But it can make sense in IoT use cases where it is more memory efficient to deliver a single CA certificate rather than a whole bundle of them. I've seen this done before.

Looking at the cert showed by the site it looks to me they tried to introduce an intermediate CA and forget to add the root CA cert in the chain, but i can be wrong

Try using the openssl s_client -showcerts -connect otau.meethue.com:443 command I used to print the certificate chain in my first comment. As of now I'm still being sent 3 certificates by the server, including the root CA certificate (CN=otau.meethue.com, CN=intermediate and CN=root). But the hostname resolves to a Google Cloud IP, so there may be a CDN in between and it's quite possible you're hitting a different server that is configured differently. In that case it would be interesting to see if the certificates at least match, so feel free to report the output of the command back to this issue.

brlodi commented 8 months ago

Can't be entirely sure.

I've never used it but I assume the official Hue hub hits this same server, correct? Have we confirmed that Signify's own app isn't running into the same cert problem? If it works we can't can't say it was definitely on purpose but if it does break in the official app it would be a strong sign that this is unintended on Signify's part.

FWIW my output from OpenSSL: (appears to match the chain in the OP)

``` ~ → openssl s_client -showcerts -connect otau.meethue.com:443 Connecting to 35.241.40.143 CONNECTED(00000005) depth=2 C=NL, O=Philips Hue, CN=root verify error:num=19:self-signed certificate in certificate chain verify return:1 depth=2 C=NL, O=Philips Hue, CN=root verify return:1 depth=1 C=NL, O=Philips Hue, CN=intermediate verify return:1 depth=0 C=NL, O=Philips Hue, CN=otau.meethue.com verify return:1 --- Certificate chain 0 s:C=NL, O=Philips Hue, CN=otau.meethue.com i:C=NL, O=Philips Hue, CN=intermediate a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Dec 28 08:19:26 2023 GMT; NotAfter: Dec 27 08:19:26 2024 GMT -----BEGIN CERTIFICATE----- MIICOzCCAeGgAwIBAgICEIYwCgYIKoZIzj0EAwIwOjELMAkGA1UEBhMCTkwxFDAS BgNVBAoMC1BoaWxpcHMgSHVlMRUwEwYDVQQDDAxpbnRlcm1lZGlhdGUwHhcNMjMx MjI4MDgxOTI2WhcNMjQxMjI3MDgxOTI2WjA+MQswCQYDVQQGEwJOTDEUMBIGA1UE ChMLUGhpbGlwcyBIdWUxGTAXBgNVBAMTEG90YXUubWVldGh1ZS5jb20wWTATBgcq hkjOPQIBBggqhkjOPQMBBwNCAASsjJWztRR+y0Fh1moAEFInY7Hm0Bs2sNVlWwbh G1ND0xM5q88rRDPU/0uhseC1Q8zrIG5odks/qKFeyWPuluvmo4HSMIHPMAwGA1Ud EwEB/wQCMAAwHQYDVR0OBBYEFODTXkF2YAGuNZdIYa9CXgVRHYg5MFsGA1UdIwRU MFKAFE5wTAXeCEvl+iOFRD6V/DHv8j+MoTakNDAyMQswCQYDVQQGEwJOTDEUMBIG A1UECgwLUGhpbGlwcyBIdWUxDTALBgNVBAMMBHJvb3SCAhAEMA4GA1UdDwEB/wQE AwIFoDAWBgNVHSUBAf8EDDAKBggrBgEFBQcDATAbBgNVHREEFDASghBvdGF1Lm1l ZXRodWUuY29tMAoGCCqGSM49BAMCA0gAMEUCICbyQzlJ3qB+hXQpK1Div1LPrCA3 4cBeGDph2y2gvJhYAiEA7R2WE6/qVsxlkL5h4Puh+yxp/ofs0laGMzy9FzQczYM= -----END CERTIFICATE----- 1 s:C=NL, O=Philips Hue, CN=intermediate i:C=NL, O=Philips Hue, CN=root a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Sep 21 09:35:35 2021 GMT; NotAfter: Nov 10 09:35:35 2026 GMT -----BEGIN CERTIFICATE----- MIIBwTCCAWigAwIBAgICEAQwCgYIKoZIzj0EAwIwMjELMAkGA1UEBhMCTkwxFDAS BgNVBAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MB4XDTIxMDkyMTA5MzUz NVoXDTI2MTExMDA5MzUzNVowOjELMAkGA1UEBhMCTkwxFDASBgNVBAoMC1BoaWxp cHMgSHVlMRUwEwYDVQQDDAxpbnRlcm1lZGlhdGUwWTATBgcqhkjOPQIBBggqhkjO PQMBBwNCAATApYP6FMJcZkAMz98aJIJ08gpXW3FT47ikdQ4KlmztyYeTyKPWf/sP wiTeAu5nr2y8Gj0jYhk/6FNenuRdV+w0o2YwZDAdBgNVHQ4EFgQUTnBMBd4IS+X6 I4VEPpX8Me/yP4wwHwYDVR0jBBgwFoAUCWQVgAJXOKupOv75xaJrwAof984wEgYD VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwCgYIKoZIzj0EAwIDRwAw RAIgMSvQr5cUypLgd752OzS8WF0r6duHV9el8R1YDHl2qJUCIHsvs0CiH6Xg/+9P k1lIs2YkrWB4faKk7/rEGnv4RAx1 -----END CERTIFICATE----- 2 s:C=NL, O=Philips Hue, CN=root i:C=NL, O=Philips Hue, CN=root a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256 v:NotBefore: Aug 25 07:59:43 2016 GMT; NotAfter: Jan 5 07:59:43 2068 GMT -----BEGIN CERTIFICATE----- MIIBwDCCAWagAwIBAgIJAJtrMkoTxs+WMAoGCCqGSM49BAMCMDIxCzAJBgNVBAYT Ak5MMRQwEgYDVQQKDAtQaGlsaXBzIEh1ZTENMAsGA1UEAwwEcm9vdDAgFw0xNjA4 MjUwNzU5NDNaGA8yMDY4MDEwNTA3NTk0M1owMjELMAkGA1UEBhMCTkwxFDASBgNV BAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MFkwEwYHKoZIzj0CAQYIKoZI zj0DAQcDQgAEENC1JOl6BxJrwCb+YK655zlM57VKFSi5OHDsmlCaF/EfTGGgU08/ JUtkCyMlHUUoYBZyzCBKXqRKkrT512evEKNjMGEwHQYDVR0OBBYEFAlkFYACVzir qTr++cWia8AKH/fOMB8GA1UdIwQYMBaAFAlkFYACVzirqTr++cWia8AKH/fOMA8G A1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA0gAMEUC IQDcGfyXaUl5hjr5YE8m2piXhMcDzHTNbO1RvGgz4r9IswIgFTTw/R85KyfIiW+E clwJRVSsq8EApeFREenCkRM0EIk= -----END CERTIFICATE----- --- Server certificate subject=C=NL, O=Philips Hue, CN=otau.meethue.com issuer=C=NL, O=Philips Hue, CN=intermediate --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits --- SSL handshake has read 1794 bytes and written 404 bytes Verification error: self-signed certificate in certificate chain --- New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 256 bit This TLS version forbids renegotiation. Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 19 (self-signed certificate in certificate chain) ```

deviantintegral commented 8 months ago

Perhaps it's possible to check the iOS or Android apps for a copy of one of the certificates. If they are intentionally self-signing, I can't believe that they wouldn't also pin the root or an intermediate certificate in the app.

kaechele commented 8 months ago

Implementation is done in principle:

Will be doing some more testing tomorrow and then prepare the PRs. But is seems to be working fine for me locally.

There are a few things I'd like to work out that would require a bit deeper refactoring of things, but that can happen over time:

The CA bundle is currently generated whenever the OTA function is first started. We should probably have a smarter way of looking whether we need to update the bundle (i.e. caching, freshness checking).
The ota.insecure_tls and ota.ca settings options only work for zigbeeOTA based devices. I haven't found a quick way to get an option from the settings to zhc.common.getAxios() in a way that it applies to all OTA implementations.

kaechele commented 8 months ago

Perhaps it's possible to check the iOS or Android apps for a copy of one of the certificates. If they are intentionally self-signing, I can't believe that they wouldn't also pin the root or an intermediate certificate in the app.

I'm pretty confident the pinning would be happening in the Hue bridge, not in the App. I don't have a Hue bridge available to me right now, so we'd need to get someone to root theirs and check what's going on there.

theDiverDK commented 8 months ago

Kaechele amazing work :)

Hope it works so I can update my single old Philips Hue bulb I found in a drawer ;-)

MichelBarneveld commented 8 months ago

My current work around until the fix is implemented is this (btw: I'm not running z2m in a container): 1) create an "extra_ca.pem" file with the public root and intermediate certificates for https://otau.meethue.com 2) create an environment vairable NODE_EXTRA_CA_CERTS with the value of the full path to the extra_ca.pem file 3) restart zigbee2mqtt

content of extra_ca.pem: -----BEGIN CERTIFICATE----- MIIBwDCCAWagAwIBAgIJAJtrMkoTxs+WMAoGCCqGSM49BAMCMDIxCzAJBgNVBAYT Ak5MMRQwEgYDVQQKDAtQaGlsaXBzIEh1ZTENMAsGA1UEAwwEcm9vdDAgFw0xNjA4 MjUwNzU5NDNaGA8yMDY4MDEwNTA3NTk0M1owMjELMAkGA1UEBhMCTkwxFDASBgNV BAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MFkwEwYHKoZIzj0CAQYIKoZI zj0DAQcDQgAEENC1JOl6BxJrwCb+YK655zlM57VKFSi5OHDsmlCaF/EfTGGgU08/ JUtkCyMlHUUoYBZyzCBKXqRKkrT512evEKNjMGEwHQYDVR0OBBYEFAlkFYACVzir qTr++cWia8AKH/fOMB8GA1UdIwQYMBaAFAlkFYACVzirqTr++cWia8AKH/fOMA8G A1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA0gAMEUC IQDcGfyXaUl5hjr5YE8m2piXhMcDzHTNbO1RvGgz4r9IswIgFTTw/R85KyfIiW+E clwJRVSsq8EApeFREenCkRM0EIk= -----END CERTIFICATE-----

oscfor commented 8 months ago

My current work around until the fix is implemented is this (btw: I'm not running z2m in a container):

create an "extra_ca.pem" file with the public root and intermediate certificates for https://otau.meethue.com

create an environment vairable NODE_EXTRA_CA_CERTS with the value of the full path to the extra_ca.pem file

restart zigbee2mqtt

content of extra_ca.pem: -----BEGIN CERTIFICATE----- MIIBwDCCAWagAwIBAgIJAJtrMkoTxs+WMAoGCCqGSM49BAMCMDIxCzAJBgNVBAYT Ak5MMRQwEgYDVQQKDAtQaGlsaXBzIEh1ZTENMAsGA1UEAwwEcm9vdDAgFw0xNjA4 MjUwNzU5NDNaGA8yMDY4MDEwNTA3NTk0M1owMjELMAkGA1UEBhMCTkwxFDASBgNV BAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MFkwEwYHKoZIzj0CAQYIKoZI zj0DAQcDQgAEENC1JOl6BxJrwCb+YK655zlM57VKFSi5OHDsmlCaF/EfTGGgU08/ JUtkCyMlHUUoYBZyzCBKXqRKkrT512evEKNjMGEwHQYDVR0OBBYEFAlkFYACVzir qTr++cWia8AKH/fOMB8GA1UdIwQYMBaAFAlkFYACVzirqTr++cWia8AKH/fOMA8G A1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA0gAMEUC IQDcGfyXaUl5hjr5YE8m2piXhMcDzHTNbO1RvGgz4r9IswIgFTTw/R85KyfIiW+E clwJRVSsq8EApeFREenCkRM0EIk= -----END CERTIFICATE-----

-----BEGIN CERTIFICATE----- MIIBwTCCAWigAwIBAgICEAQwCgYIKoZIzj0EAwIwMjELMAkGA1UEBhMCTkwxFDAS BgNVBAoMC1BoaWxpcHMgSHVlMQ0wCwYDVQQDDARyb290MB4XDTIxMDkyMTA5MzUz NVoXDTI2MTExMDA5MzUzNVowOjELMAkGA1UEBhMCTkwxFDASBgNVBAoMC1BoaWxp cHMgSHVlMRUwEwYDVQQDDAxpbnRlcm1lZGlhdGUwWTATBgcqhkjOPQIBBggqhkjO PQMBBwNCAATApYP6FMJcZkAMz98aJIJ08gpXW3FT47ikdQ4KlmztyYeTyKPWf/sP wiTeAu5nr2y8Gj0jYhk/6FNenuRdV+w0o2YwZDAdBgNVHQ4EFgQUTnBMBd4IS+X6 I4VEPpX8Me/yP4wwHwYDVR0jBBgwFoAUCWQVgAJXOKupOv75xaJrwAof984wEgYD VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwCgYIKoZIzj0EAwIDRwAw RAIgMSvQr5cUypLgd752OzS8WF0r6duHV9el8R1YDHl2qJUCIHsvs0CiH6Xg/+9P k1lIs2YkrWB4faKk7/rEGnv4RAx1 -----END CERTIFICATE-----

Great workaround. Thanks!

Now I just need to find out how to add an environment variable to z2m...

MichelBarneveld commented 8 months ago

@oscfor It's an environment variable on Operating System level, not specific for z2m or nodeJS. Here is a link how to set an environment variable permanently on different operating systems: https://www3.ntu.edu.sg/home/ehchua/programming/howto/Environment_Variables.html

deviantintegral commented 8 months ago

My current work around until the fix is implemented is this (btw: I'm not running z2m in a container):

I took a different approach and downloaded the firmware manually, and then applied them using a local OTA index: https://www.zigbee2mqtt.io/guide/usage/ota_updates.html#local-ota-index-and-firmware-files

However, your method doesn't require finding the right JSON definition for the firmware file, which would be a pain if you had many different devices to update. Nice! 👏

adrienneo19 commented 8 months ago

I've just tried with haos but it doesn't work, probably something I'm doing wrong. i put the extra_ca.pem file in the ssl folder folder and then in configuration.yaml with the following addons:

https://github.com/Athozs/hass-environment-variable https://github.com/Athozs/hass-environment-variable environment_variable: NODE_EXTRA_CA_CERTS: /ssl/extra_ca.pem

environment_variable:
  NODE_EXTRA_CA_CERTS: /ssl/extra_ca.pem

unfortunately it doesn't work. if you have an idea :)

jackwilsdon commented 8 months ago

@adrienneo19 that custom component sets environment variables for Home Assistant itself, not its addons. I don't think there's a way to set environment variables for addons - you're probably best waiting until the issue is fixed in Z2M.

Koenkk commented 8 months ago

This is solved now, many thanks @kaechele ❤️

Changes will be available in the dev branch in a few hours from now.

LetThatSinkIn commented 8 months ago

This is solved now, many thanks @kaechele ❤️

Changes will be available in the dev branch in a few hours from now.

Is it possible to switch to the dev branch if using Z2M as an add-on in HA?

dueyfinster commented 8 months ago

This is solved now, many thanks @kaechele ❤️ Changes will be available in the dev branch in a few hours from now.

Is it possible to switch to the dev branch if using Z2M as an add-on in HA?

I'm on stable branch - updated now and it's working away upgrading the firmware on a hue bulb I have.

LetThatSinkIn commented 8 months ago

This is solved now, many thanks @kaechele ❤️ Changes will be available in the dev branch in a few hours from now.

Is it possible to switch to the dev branch if using Z2M as an add-on in HA?

I'm on stable branch - updated now and it's working away upgrading the firmware on a hue bulb I have.

Oh, you're right, I looked before and didn't see that available. But sure enough, there it is. THANK YOU!

sippeeey commented 8 months ago

How does it actually work?

I'm on version 1.35.1 with zigbee2mqtt LXC on proxmox.

At the moment the firmware upgrade on my two phillips lightstrips still failed. first one the update starts with 0% an then failed with the message: "aborted by device" now it says only one of my two devices has an update, but when i start this it says: "no new image available"

kaechele commented 8 months ago

That looks more like #400 than anything to do with this particular bug, which is fixed in recent versions.

kaechele commented 8 months ago

Fun fact: the certificate was changed back to a proper DigiCert signed one recently (certificate issued January 9). So it seems the change was likely unintentional.

Koenkk / zigbee-OTA

`https://otau.meethue.com` started using a self-signed certificate #420