Closed samuelclay closed 4 months ago
KaeLL
commented
4 months ago I had this issue too back in March. Ended up switching the esp crt bundle from minimal to full. The emergency update server bring-up was done by a coworker, and crisis was averted.
samuelclay
commented
4 months ago We're already using the full default certificate bundle, but what was the emergency update server bring-up? Does that mean you added certificates to the chain on your server? Which ones and how did you know which ones?
KaeLL
commented
4 months ago @windol
samuelclay
commented
4 months ago Solved the problem. I'd like to pay it forward here to anybody else who may come across this issue in the future. Here's what I did.
I built a minimal test case that (included below):
That allowed me to narrow down the issue. At first it worked. Then I imported my project's sdkconfig.defaults and adjusted a few of the configuration options so that it compiled (mainly about partition maps needing to change to address the larger flash size). And then I was able to reproduce the issue.
There were about 250 differences between the two sdkconfig settings. I used a binary search to narrow it down to a single setting: CONFIG_MBEDTLS_SHA512_C: Enable the SHA-384 and SHA-512 cryptographic hash algorithms
We had that turned off in our shipping code, while it's turned on by default. Switching it on got everything working again. Obviously I can't switch it on because these devices are out in the field and I can't change their code. But that led me down the path of figuring out what changes were needed in the SSL certificate.
I learned that our certificate was using a signature algorithm of ECDSA SHA-384. By creating a RSA 2048-bit certificate signing request (CSR) and using that CSR when applying for a SSL certificate, I could generate a certification that used RSA SHA-256, which would be supported by my devices using # CONFIG_MBEDTLS_SHA512_C is not set.
samuelclay
commented
4 months ago Here's esp32_https_test.c, my minimal test case:
#include "esp_crt_bundle.h"
#include "esp_event.h"
#include "esp_http_client.h"
#include "esp_log.h"
#include "esp_netif.h"
#include "esp_system.h"
#include "esp_tls.h"
#include "esp_wifi.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "nvs_flash.h"
#include <stdio.h>
static const char *TAG = "HTTPS_TEST";
#define WIFI_POWER_19_5dBm 78
#define DEFAULT_SCAN_LIST_SIZE 10
#define WIFI_SSID "<snip>"
#define WIFI_PASS "<snip>"
esp_err_t _http_event_handler(esp_http_client_event_t *evt) {
switch (evt->event_id) {
case HTTP_EVENT_ERROR:
ESP_LOGI(TAG, "HTTP_EVENT_ERROR");
break;
case HTTP_EVENT_ON_CONNECTED:
ESP_LOGI(TAG, "HTTP_EVENT_ON_CONNECTED");
break;
case HTTP_EVENT_HEADER_SENT:
ESP_LOGI(TAG, "HTTP_EVENT_HEADER_SENT");
break;
case HTTP_EVENT_ON_HEADER:
ESP_LOGI(TAG, "HTTP_EVENT_ON_HEADER, key=%s, value=%s", evt->header_key,
evt->header_value);
break;
case HTTP_EVENT_ON_DATA:
ESP_LOGI(TAG, "HTTP_EVENT_ON_DATA, len=%d", evt->data_len);
if (!esp_http_client_is_chunked_response(evt->client)) {
printf("%.*s", evt->data_len, (char *)evt->data);
}
break;
case HTTP_EVENT_ON_FINISH:
ESP_LOGI(TAG, "HTTP_EVENT_ON_FINISH");
break;
case HTTP_EVENT_DISCONNECTED:
ESP_LOGI(TAG, "HTTP_EVENT_DISCONNECTED");
break;
default:
ESP_LOGI(TAG, "HTTP_EVENT_UNKNOWN, event=%d", evt->event_id);
break;
}
return ESP_OK;
}
void create_client_and_test_website() {
esp_http_client_config_t config = {
.url = "https://example.com",
.event_handler = _http_event_handler,
.method = HTTP_METHOD_GET,
.timeout_ms = 30000,
.cert_pem = NULL,
.crt_bundle_attach = esp_crt_bundle_attach,
};
esp_http_client_handle_t client = esp_http_client_init(&config);
// Perform the HTTP GET request
esp_err_t err = esp_http_client_perform(client);
if (err == ESP_OK) {
ESP_LOGI(TAG, "HTTP GET Status = %d, content_length = %" PRIi64,
esp_http_client_get_status_code(client),
esp_http_client_get_content_length(client));
} else {
ESP_LOGE(TAG, "HTTP GET request failed: %s", esp_err_to_name(err));
}
esp_http_client_cleanup(client);
}
void wifi_scan() {
uint16_t number = DEFAULT_SCAN_LIST_SIZE;
wifi_ap_record_t ap_info[DEFAULT_SCAN_LIST_SIZE];
uint16_t ap_count = 0;
memset(ap_info, 0, sizeof(ap_info));
ESP_ERROR_CHECK(esp_wifi_scan_start(NULL, true));
ESP_ERROR_CHECK(esp_wifi_scan_get_ap_records(&number, ap_info));
ESP_ERROR_CHECK(esp_wifi_scan_get_ap_num(&ap_count));
ESP_LOGI(TAG, "Total APs scanned = %u", ap_count);
for (int i = 0; (i < DEFAULT_SCAN_LIST_SIZE) && (i < ap_count); i++) {
ESP_LOGI(TAG, "SSID \t\t%s", ap_info[i].ssid);
ESP_LOGI(TAG, "RSSI \t\t%d", ap_info[i].rssi);
ESP_LOGI(TAG, "Channel \t\t%d\n", ap_info[i].primary);
}
}
static void wifi_event_handler(void *arg, esp_event_base_t event_base,
int32_t event_id, void *event_data) {
if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
int8_t power = 0;
esp_wifi_get_max_tx_power(&power);
ESP_LOGI(TAG, "WiFi TX Power was %d", power);
esp_wifi_set_max_tx_power(WIFI_POWER_19_5dBm);
esp_wifi_get_max_tx_power(&power);
ESP_LOGI(TAG, "WiFi TX Power is now %d", power);
ESP_LOGI(TAG, "Connecting to WiFi...");
esp_wifi_connect();
} else if (event_base == WIFI_EVENT &&
event_id == WIFI_EVENT_STA_DISCONNECTED) {
wifi_event_sta_disconnected_t *disconnected =
(wifi_event_sta_disconnected_t *)event_data;
ESP_LOGI(TAG, "Disconnected from WiFi, reason: %d", disconnected->reason);
switch (disconnected->reason) {
case WIFI_REASON_AUTH_EXPIRE:
ESP_LOGI(TAG, "Auth Expire");
break;
case WIFI_REASON_AUTH_FAIL:
ESP_LOGI(TAG, "Authentication Failed, Check your SSID and Password");
break;
case WIFI_REASON_NO_AP_FOUND:
ESP_LOGI(TAG, "AP not found, Check if the AP is available");
break;
case WIFI_REASON_BEACON_TIMEOUT:
ESP_LOGI(TAG, "Beacon Timeout");
break;
case WIFI_REASON_ASSOC_LEAVE:
ESP_LOGI(TAG, "Association Leave");
break;
case WIFI_REASON_HANDSHAKE_TIMEOUT:
ESP_LOGI(TAG, "Handshake Timeout");
break;
default:
ESP_LOGI(TAG, "WiFi disconnected, reconnecting...");
break;
}
// Sleep 5 seconds before retrying
vTaskDelay(5000 / portTICK_PERIOD_MS);
wifi_scan();
esp_wifi_connect();
} else {
ESP_LOGI(TAG, "Unknown WiFi event: %ld", event_id);
}
}
static void ip_event_handler(void *arg, esp_event_base_t event_base,
int32_t event_id, void *event_data) {
if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
ip_event_got_ip_t *event = (ip_event_got_ip_t *)event_data;
ESP_LOGI(TAG, "Got IP: " IPSTR, IP2STR(&event->ip_info.ip));
create_client_and_test_website();
} else if (event_base == WIFI_EVENT &&
event_id == WIFI_EVENT_STA_DISCONNECTED) {
wifi_event_sta_disconnected_t *disconnected =
(wifi_event_sta_disconnected_t *)event_data;
ESP_LOGI(TAG, "Disconnected from WiFi, reason: %d", disconnected->reason);
switch (disconnected->reason) {
case WIFI_REASON_AUTH_EXPIRE:
ESP_LOGI(TAG, "Auth Expire");
break;
case WIFI_REASON_AUTH_FAIL:
ESP_LOGI(TAG, "Authentication Failed, Check your SSID and Password");
break;
case WIFI_REASON_NO_AP_FOUND:
ESP_LOGI(TAG, "AP not found, Check if the AP is available");
break;
case WIFI_REASON_BEACON_TIMEOUT:
ESP_LOGI(TAG, "Beacon Timeout");
break;
case WIFI_REASON_ASSOC_LEAVE:
ESP_LOGI(TAG, "Association Leave");
break;
case WIFI_REASON_HANDSHAKE_TIMEOUT:
ESP_LOGI(TAG, "Handshake Timeout");
break;
default:
ESP_LOGI(TAG, "WiFi disconnected, reconnecting...");
break;
}
// Sleep 5 seconds before retrying
vTaskDelay(5000 / portTICK_PERIOD_MS);
wifi_scan();
esp_wifi_connect();
} else if (event_base == IP_EVENT && event_id == IP_EVENT_STA_GOT_IP) {
ip_event_got_ip_t *event = (ip_event_got_ip_t *)event_data;
ESP_LOGI(TAG, "Got IP: " IPSTR, IP2STR(&event->ip_info.ip));
create_client_and_test_website();
} else {
ESP_LOGI(TAG, "Unknown IP event: %ld", event_id);
}
}
void wifi_init() {
ESP_ERROR_CHECK(esp_netif_init());
ESP_ERROR_CHECK(esp_event_loop_create_default());
esp_netif_create_default_wifi_sta();
wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
cfg.static_tx_buf_num = 0;
cfg.dynamic_tx_buf_num = 32;
cfg.tx_buf_type = 1;
cfg.cache_tx_buf_num = 1;
cfg.static_rx_buf_num = 4;
cfg.dynamic_rx_buf_num = 32;
ESP_ERROR_CHECK(esp_wifi_init(&cfg));
esp_event_handler_instance_t instance_any_id;
esp_event_handler_instance_t instance_got_ip;
ESP_ERROR_CHECK(esp_event_handler_instance_register(
WIFI_EVENT, ESP_EVENT_ANY_ID, &wifi_event_handler, NULL,
&instance_any_id));
ESP_ERROR_CHECK(esp_event_handler_instance_register(
IP_EVENT, ESP_EVENT_ANY_ID, &ip_event_handler, NULL, &instance_got_ip));
wifi_config_t wifi_config = {
.sta =
{
.ssid = WIFI_SSID,
.password = WIFI_PASS,
.threshold.authmode = WIFI_AUTH_WPA2_PSK,
.sae_pwe_h2e = WPA3_SAE_PWE_BOTH,
.sae_h2e_identifier = "",
.pmf_cfg = {.capable = true, .required = false},
},
};
ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
esp_wifi_set_protocol(WIFI_IF_STA, WIFI_PROTOCOL_11B | WIFI_PROTOCOL_11G |
WIFI_PROTOCOL_11N);
ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
vTaskDelay(1000 / portTICK_PERIOD_MS);
ESP_ERROR_CHECK(esp_wifi_start());
vTaskDelay(1000 / portTICK_PERIOD_MS);
ESP_ERROR_CHECK(esp_wifi_set_ps(WIFI_PS_NONE));
}
void app_main(void) {
esp_log_level_set("wifi", ESP_LOG_VERBOSE);
esp_log_level_set("wifi_init", ESP_LOG_VERBOSE);
esp_err_t ret = nvs_flash_init();
if (ret == ESP_ERR_NVS_NO_FREE_PAGES ||
ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
ESP_ERROR_CHECK(nvs_flash_erase());
ret = nvs_flash_init();
}
ESP_ERROR_CHECK(ret);
wifi_init();
}
Answers checklist.
IDF version.
v5.1
Espressif SoC revision.
ESP32-S3
Operating System used.
macOS
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
Development Kit.
Custom Board
Power Supply used.
USB
What is the expected behavior?
I'm encountering an SSL handshake failure with my ESP32 devices (using ESP-IDF v5.1 with mbedTLS 3.3.0) when connecting to my server. This issue started occurring after the server's Let's Encrypt SSL certificate was renewed. When I upgrade to ESP-IDF v5.1.4, which uses mbedTLS 3.5.2, everything works as expected. But the ESP-IDF v5.1 devices are deployed in the field, so I can't easily update their firmware.
What is the actual behavior?
The ESP32 devices were able to connect successfully until a recent renewal of the SSL certificate on the server. The renewed certificate now causes the ESP32 devices to fail the connection with the following error:
The SSL certificates have an A rating on Qualys SSL Server Test and work fine on web browsers and IDF v5.1.4, just not on IDF v5.1. The server is serving a TLS 1.2 certificate with many ciphers, most of which IDF 5.1 should be able to handle.
I've confirmed that the ESP32 devices are successfully connecting to WiFi. And I verified that upgrading to a later version of mbedTLS on a test device resolves the issue (not feasible for deployed devices). I also reviewed Caddy configuration and SSL settings and switched to nginx with similar configurations, just to rule out the web server. Both Caddy and nginx provide A ratings for the SSL certificate, and work in a browser and on IDF 5.1.4.
Steps to reproduce.
Debug Logs.
Server Log (Caddy):