IDF 5.0 cannot connect to Google MQTT server (IDFGH-8568)

[Nick-tecpal]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0-beta1-742-g7bd5af7f1e-dirt

Operating System used.

Linux

How did you build your project?

If you are using Windows, please specify command line type.

Development Kit.

ESP-S3-DevkitC-1 V1.1

Power Supply used.

USB

What is the expected behavior?

IDF 5.0 can connect to our Google MQTT server.

What is the actual behavior?

Our server return 34 ，  /* @cond Internal. Numeric code: 34 / IOTC_MQTT_NOT_AUTHORIZED, /* @endcond /

Steps to reproduce.

Debug Logs.

I (12144) Tecpal net decode: [Heap: 5898836]
upgrade notify app net result -> ({"uri":"/wifi status",
. "seg": 2, "result": 4, "params": ("ssid": "TGI-WiFi-VPN", "rssi": -58, "ipAddr": "192.168.50.19"}}]
I (12654) Tepal Giot fml: [Heap: 5892036]mqtt private key [-----BEGIN EC PRIVATE KEY ---
-----END EC PRIVATE KEY- ----]
I (12664) Tepal Giot fml: (Heap: 5891888]matt connect info proiects/tecpal-dev/locations/asia-eastl/registries/cp-iot-registry-asia-sit/devices/prA1F362661
E
(14354) Tecpal Giot fml: [Heap:5891036][../components/tecpal sdk/FML/module/tecpal sdk giot.c:142]->on connection state changed cbt}
MOTT Open Failed with reason[34].
I
(14354) Tepal Giot fml:
[Heap: 5891084]
iot core ready to closed
E (14364) Tecpal Giot
fml.
[Hean:5892274
tecpal
sdk/FML/module/tecpal sdk aiot.c:2291->iot matt
task handlerf1 • Tor Core Delete fol..

More Information.

No response

[euripedesrocha]

Hi @Nick-tecpal, could you clarify on which library you are using for MQTT client?

[Jiven-tecpal]

The tracking discovery is caused by IDF5.0 using the latest mbedtls library,Thus causing the JWT token signature exception

[Jiven-tecpal]

google iot connect info: client id:projects/tecpal-dev/locations/asia-east1/registries/cp-iot-registry-asia-sit/devices/PC_A1E3626E jwt:eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NjcyNzY3OTcsImV4cCI6MTY2NzMxOTk5NywiYXVkIjoidGVjcGFsLWRldiJ9.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==

[Jiven-tecpal]

@euripedesrocha please help

[Jiven-tecpal]

google iot repo commit id:758425b9c2ec82d35a0af77e31dcd9b69c303f8a (HEAD -> master, origin/master, origin/HEAD) @euripedesrocha

[euripedesrocha]

Hi @jiven57316 is this related to @Nick-tecpal original issue? If not, could you please open a new issue describing your problem.

[Jiven-tecpal]

“Hi @jiven57316 is this related to @Nick-tecpal original issue? If not, could you please open a new issue describing your problem.”-----Yes,We are in the same company and i working on this issue. @euripedesrocha

[Nick-tecpal]

Hi @euripedesrocha , we're trying to figure out the root cause. It seems the iotc_bsp_ecc function is not working in IDF 5.0 which is using the latest mbedtls library.

[zztiger123]

update: We evaluated this issue further and we observed that crypto and TLS layers from Google IoT SDK would need careful migration to mbedTLS-3.x. Please find migration guide available here. Please file an issue on upstream repository if you need further help on this topic. From IDF side, we do not see any issue as such and hence we would close it. Thank you.

[mahavirj]

Since this change is from the external component as highlighted in the earlier comment, closing it from IDF side.

[Nick-tecpal]

Hi, we have found a new issue related to the mbedtls V3.2.1.

[Nick-tecpal]

issue:When switching to IDF5.0, the JWT generated before the MQTT connection fails to generate a valid signature, so the value of AAAAA appears and an error code of 34 is returned.

chip:ESP32S3N8R8

idf-version:885e501(HEAD -> v5.0, origin/release/v5.0)

mbedtls version:7b428b1bf260ce1fec4e3ffb6494070d439a3a67 components/mbedtls/mbedtls (mbedtls-3.2.1-6-g7b428b1bf)

google mqtt info:

I (10263) Tecpal_Giot_fml: [Heap:7620888]mqtt connect info [projects/tecpal-dev/locations/asia-east1/registries/cp-iot-registry-asia-sit/devices/PC_A1E3626E],JWT[eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NzU4Mzg3NTcsImV4cCI6MTY3NTg4MTk1NywiYXVkIjoidGVjcGFsLWRldiJ9.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==],state:0

Correct jwt token:

"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NjcyODUwMzYsImV4cCI6MTY2NzMyODIzNiwiYXVkIjoidGVjcGFsLWRldiJ9.NmmMPO9_5VBC7adsWCYAFk8-Vq2Z6tw2sAsd-PVeWCNpXIGZ361buLUWEafe6ZoDnNzhwyroPGCkD_yzc4LTiA=="

Actual jwt token:

"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NzU4Mzg3NTcsImV4cCI6MTY3NTg4MTk1NywiYXVkIjoidGVjcGFsLWRldiJ9.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="

Related source code from application layer：

static void iot_mqtt_task_handler(void pvParameters) { security_key_t security_key = tecpal_sdk_system_get_security_key(); iotc_crypto_key_data_t iotc_connect_private_key_data; iotc_connect_private_key_data.crypto_key_signature_algorithm = IOTC_CRYPTO_KEY_SIGNATURE_ALGORITHM_ES256; iotc_connect_private_key_data.crypto_key_union_type = IOTC_CRYPTO_KEY_UNION_TYPE_PEM; //iotc_connect_private_key_data.crypto_key_union.key_pem.key = (char )private_key; iotc_connect_private_key_data.crypto_key_union.key_pem.key = (char )security_key->mqtts_ca_cert.pdata.data;

iotc_state_t iot_state; 
iot_state = iotc_initialize(); 
if (IOTC_STATE_OK != iot_state) { 
    GIOT_FML_LOGE(" iotc failed to initialize, error: %d\n", iot_state); 
    is_iot_open = false; 
    iotc_shutdown(); 
    vTaskDelete(mqtt_task_handle); 
} 
iotc_handle = iotc_create_context(); 
if (IOTC_INVALID_CONTEXT_HANDLE >= iotc_handle) { 
    GIOT_FML_LOGE(" iotc failed to create context, error: %d\n", -iotc_handle); 
    iotc_shutdown(); 

    is_iot_open = false; 
    vTaskDelete(mqtt_task_handle); 
}

/ Generate the client authentication JWT, which will serve as the MQTT password. / size_t bytes_written = 0; iot_state = iotc_create_iotcore_jwt( CONFIG_GIOT_PROJECT_ID, /jwt_expiration_period_sec=/12*3600, &iotc_connect_private_key_data, jwt, IOTC_JWT_SIZE, &bytes_written); GIOT_FML_LOGI("mqtt private key [%s],jwt[%s],len:%d",iotc_connect_private_key_data.crypto_key_union.key_pem.key,jwt,IOTC_JWT_SIZE); //strcpy(jwt,"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NjcyODUwMzYsImV4cCI6MTY2NzMyODIzNiwiYXVkIjoidGVjcGFsLWRldiJ9.NmmMPO9_5VBC7adsWCYAFk8-Vq2Z6tw2sAsd-PVeWCNpXIGZ361buLUWEafe6ZoDnNzhwyroPGCkD_yzc4LTiA==");
//use this jwt can solve the issue if (IOTC_STATE_OK != iot_state) { GIOT_FML_LOGE("iotc_create_iotcore_jwt returned with error: %ul", iot_state); // sdk_event_center_notify_evt(NTP_TIME_SYNC,NULL,0); is_iot_open = false; iotc_delete_context(iotc_handle); iotc_handle = IOTC_INVALID_CONTEXT_HANDLE; vTaskDelete(mqtt_task_handle); }

iot_state = iot_mqtt_connect(); 
if (IOTC_STATE_OK != iot_state) { 
    GIOT_FML_LOGE("iotc_connect fail with ret:[%d]", iot_state); 
    vTaskDelete(mqtt_task_handle); 
} 
iotc_events_process_blocking(); 
GIOT_FML_LOGI("\r\n \t iot core ready to closed ............................"); 

if(mqtt_task_handle){ 
    iotc_delete_context(iotc_handle); 
    iotc_shutdown(); 
    iotc_handle = IOTC_INVALID_CONTEXT_HANDLE; 
    GIOT_FML_LOGE("IOT Core Delete [%d]....\r\n",is_reconnect_fail); 
    is_iot_open = false; 
    mqtt_task_handle = NULL; 
    if(is_reconnect_fail){ 
        is_reconnect_fail = false; 
        sdk_event_center_notify_evt(NTP_TIME_SYNC ,NULL,0); 
    } 
    vTaskDelete(mqtt_task_handle); 
} 

}

// Related driven code from Google IOT： // create the JWT: b64(h).b64(p).b64(ecc(sha256(b64(h).b64(p)))) // h = header // p = payload // b64 = base64 // sha = Secure Hash Algorithm // ecc = Elliptic Curve Cryptography iotc_state_t iotc_create_iotcore_jwt( const char project_id, uint32_t expiration_period_sec, const iotc_crypto_key_data_t private_key_data, char dst_jwt_buf, size_t dst_jwt_buf_len, size_t bytes_written) { if (NULL == project_id || NULL == private_key_data || NULL == dst_jwt_buf || NULL == bytes_written) { return IOTC_INVALID_PARAMETER; }

if (IOTC_CRYPTO_KEY_SIGNATURE_ALGORITHM_ES256 != private_key_data->crypto_key_signature_algorithm) { return IOTC_ALG_NOT_SUPPORTED_ERROR; }

switch (private_key_data->crypto_key_union_type) { case IOTC_CRYPTO_KEY_UNION_TYPE_PEM: if (NULL == private_key_data->crypto_key_union.key_pem.key) { return IOTC_NULL_KEY_DATA_ERROR; } break; case IOTC_CRYPTO_KEY_UNION_TYPE_SLOT_ID: case IOTC_CRYPTO_KEY_UNION_TYPE_CUSTOM: / it's a valid scenario that the custom data could be null, if the key is hard coded in the BSP / break; default: return IOTC_NOT_IMPLEMENTED; }

if (IOTC_JWT_PROJECTID_MAX_LEN < strlen(project_id)) { *bytes_written = IOTC_JWT_PROJECTID_MAX_LEN; return IOTC_JWT_PROJECTID_TOO_LONG_ERROR; }

iotc_bsp_crypto_state_t ret = IOTC_BSP_CRYPTO_ERROR;

// create base64 encoded header and payload: b64(h).b64(p) IOTC_CHECK_CRYPTO(ret = _iotc_create_iotcore_jwt_b64h_b64p( (unsigned char*)dst_jwt_buf, dst_jwt_buf_len, bytes_written, project_id, expiration_period_sec, "ES256"));

// create sha256 hash of b64(h).b64(p): sha256(b64(h).b64(p)) uint8_t sha256_b64h_b64p[32] = {0}; IOTC_CHECK_CRYPTO(ret = iotc_bsp_sha256(sha256_b64h_b64p, (const uint8_t)dst_jwt_buf, bytes_written));

// add second dot, separating b64(h).b64(p) and b64(eccsignature) dst_jwt_buf[bytes_written] = '.'; ++bytes_written;

// create ecc signature: ecc(sha256(b64(h).b64(p))) size_t bytes_written_ecc_signature = 0; uint8_t ecc_signature[IOTC_JWT_MAX_SIGNATURE_SIZE] = {0}; IOTC_CHECK_CRYPTO(ret = iotc_bsp_ecc(private_key_data, ecc_signature, IOTC_JWT_MAX_SIGNATURE_SIZE, &bytes_written_ecc_signature, sha256_b64h_b64p, 32));

// base64 encode the ecc signature size_t bytes_written_ecc_signature_base64 = 0; ret = iotc_bsp_base64_encode_urlsafe( (unsigned char)dst_jwt_buf + bytes_written, dst_jwt_buf_len - *bytes_written, &bytes_written_ecc_signature_base64, ecc_signature, bytes_written_ecc_signature);

*bytes_written += bytes_written_ecc_signature_base64;

IOTC_CHECK_CRYPTO(ret);

return IOTC_STATE_OK;

err_handling:

switch (ret) { case IOTC_BSP_CRYPTO_STATE_OK: return IOTC_STATE_OK; case IOTC_BSP_CRYPTO_BUFFER_TOO_SMALL_ERROR: return IOTC_BUFFER_TOO_SMALL_ERROR; case IOTC_BSP_CRYPTO_INVALID_INPUT_PARAMETER_ERROR: return IOTC_INVALID_PARAMETER; default: return IOTC_JWT_FORMATTION_ERROR; }

return IOTC_JWT_FORMATTION_ERROR; }

[Nick-tecpal]

Archive.zip

[Nick-tecpal]

MQTT issue on IDF5.0.docx