AFR & ESP32: Run-time configuration of Wi-Fi credentials (Wi-Fi Provisioning) for a Device

[dipen-1337lab]

Namaste Community Members,

Our application is developed using FreeRTOS V202002.00 and ESP32-DevKitC target board.

Below is the thought process for provisioning the run-time configuration of the Wi-Fi credentials within the ESP32 device at every power ON and, as part of the Device setup.

1. On Device boot, check for available Wi-Fi credentials.
2. Try connecting to the Wi-Fi network using available credentials, if any.
3. If device cannot connect to Wi-Fi network or, Wi-Fi network credentials not available with the device, then put the device in AP mode.
4. User should be able to connect to device using predefined SSID and Password.
5. User should be able to browse a webpage hosted on device under AP mode.
6. Webpage hosted on device, should accept Wi-Fi credentials and User's credentials too (User authentication is managed by our application once internet access becomes available).
7. Device should switch to Station mode and, try connecting to an Access Point using the new Wi-Fi credentials received.
8. If device is able to connect to the Internet, it should turn OFF the AP mode.
9. If device is not able to connect to the Internet, it should display an error message on the web-page and, remove the newly saved Wi-Fi credentials from persistent memory. Then, the device should resume from step 3 or, restart and resume from step 1.
10. Once device is correctly configured, it should proceed with firmware initialisation process.

Is this a feasible approach for configuring new devices as part of the site installation process? Also, it would cater to scenarios where the Wi-Fi SSID gets modified. If Yes, kindly suggest reference links to work on the implementation and, integration within our FreeRTOS V202002.00 based application. If not, suggestions as part of a-better-approach-should-be, are more than welcome!

PS. I have gone through the Mobile SDKs for FreeRTOS Bluetooth devices for an alternate approach.

Thanks | Regards, Dipen

[aggarw13]

Namaste @dipen-1337lab, the state sequence you have shared for supporting WiFi provisioning in softAPI mode at power up for your application seems logical.

However, the V202002.00 release does not provide an ESP-IDF component or example showing softAP based WiFi provisioning. The ESP-IDF SDK in the V202007.00 and subsequent releases does contain a "wifi_provisioning" component) that supports both BLE and softAP based WiFi provisioning. The latest ESP-IDF v4.2 release also provides examples and documentation of using the "wifi_provisioning" component to create either a softAP or BLE based WiFi provisioning application here. As the API of the component has not changed in the SDK between the FreeRTOS V202007.00 release and the latest ESP-IDF v4.2, the examples can be used as reference for using the component library to build the the functionality.

However, for utilizing the component, you may need to consider upgrading your application to at least the V202007.00 release.

Alternatively, @shubhamkulkarni97, can you provide more insight about whether the ESP-IDF SDK can be upgraded in the 202002.00 release to use the SDK supplied in the 202007.00 version so that WiFI provisioning component can be used?

[dipen-1337lab]

Namaste @aggarw13,

Appreciate for your prompt response on my requirement. Thank you! I will check out the references, which you have shared.

With regards to the implementation of the Wi-Fi provisioning on V202002.00 release, my initial thought is, this would have to be done on similar lines to the coreHTTP based Mutual Authentication support included in our application, which was quite cumbersome.

Meanwhile, let's await insights from @shubhamkulkarni97.

Thanks | Regards, Dipen

[aggarw13]

Namaster @dipen-1337lab,

With regards to the implementation of the Wi-Fi provisioning on V202002.00 release, my initial thought is, this would have to be done on similar lines to the coreHTTP based Mutual Authentication support included in our application, which was quite cumbersome.

Both the newer coreHTTP library (that is provided in the 202012.00 release) and the old HTTP library (that is part of the 202002.00 release) will not be able to provide functionality of a HTTP server as these are only HTTP client libraries. You will need to utilize an HTTP library that supports setting up a server on the device.

As mentioned before, if you're able to migrate to the newer ESP-IDF version (at least the SDK in 202007.00 version), my suggestion will be to reference the the sofAP example provided here that provides your desired functionality of setting up an HTTPD web server on the device for being able to service a mobile application based client for accepting WiFi SSID and password information.

I have found an example here by @shubhamkulkarni97 that implements softAP based WiFi provisioning over the 2020202.00 release using the ESP-IDF wifi_provisioning and protocomm components 🙂. The last 2 commits in the branch show how you can migrate to the ESP-IDF SDK as well as add a WiFi provisioning app.

Hope that helps!

[shubhamkulkarni97]

Hi @dipen-1337lab @aggarw13,

ESP-IDF SDK provided in version 202002.00 supports wifi_provisioning and protocomm components. Just because of dependency of these components on LwIP networking stack, we had not included it in esp-afr-sdk.

As @aggarw13 mentioned, you can refer to example in my fork of amazon-freertos to implement wifi_provisioning in your application. Please find steps mentioned in this issue, to verify SoftAP provisioning.

Feel free to post if you face any issues with the changes.

Thanks, Shubham

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

Appreciate your prompt responses on this issue, especially considering the fact that you guys are primarily occupied with your respective assignments to work on. Thank you, guys!

I've gone through all the links shared above and, looks like I'm sorted to go ahead with the Wi-Fi provisioning integration in our application code. I shall revert if need be, as I progress.

Thanks | Regards, Dipen

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

Please find steps mentioned in this issue, to verify SoftAP provisioning.

Just sharing an interim update that I could successfully test the SoftAP Provisioning demo code using the ESP SoftAP Provisioning Mobile App. An ESP32 DevKitC device running the SoftAP Provisioning demo firmware successfully connected to my home Router (Access Point).

Some logs are shared below

I (394) app: SoftAP started
I (404) app_prov: SoftAP Provisioning started with SSID 'PROV_DFAF9C', Password 'abcd1234'
I (21784) wifi: new:<1,1>, old:<1,0>, ap:<1,1>, sta:<255,255>, prof:1
I (21784) wifi: station: 96:XX:93:XX:2d:XX join, AID=1, bgn, 40U
I (21814) app: station:96:XX:93:XX:2d:XX join, AID=1
I (22014) tcpip_adapter: softAP assign IP to station,IP is: 192.168.4.2
I (109244) app_prov_handler: WiFi Credentials Received :
        ssid <SSID>
        password <PWD>
I (109304) wifi: mode : sta (4c:XX:ae:XX:af:XX) + softAP (4c:XX:ae:XX:af:XX)
I (109304) app_prov: STA Start
I (109314) app_prov_handler: WiFi Credentials Applied
I (109434) wifi: new:<1,1>, old:<1,1>, ap:<1,1>, sta:<1,0>, prof:1
I (110424) wifi: state: init -> auth (b0)
I (110424) wifi: state: auth -> assoc (0)
I (110434) wifi: state: assoc -> run (10)
I (110544) wifi: connected with <SSID>, channel 1, BW20, bssid = 14:XX:85:XX:45:XX
I (110544) wifi: pm start, type: 1

Further, up on restarting the device, the ESP32 DevKitC board directly connected to the Access Point configured in the above step.

I (284) app_prov: Found ssid <SSID>
I (294) app_prov: Found password <PWD>
I (294) app: Starting WiFi station
I (384) phy: phy_version: 4102, 2fa7a43, Jul 15 2019, 13:06:06, 0, 0
I (384) wifi: mode : sta (4c:XX:ae:XX:af:XX)
I (504) wifi: new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (504) wifi: state: init -> auth (b0)
I (514) wifi: state: auth -> assoc (0)
I (514) wifi: state: assoc -> run (10)
I (624) wifi: connected with <SSID>, channel 1, BW20, bssid = 14:XX:85:XX:45:XX
I (634) wifi: pm start, type: 1

I (1284) event: sta ip: 192.168.19.49, mask: 255.255.255.0, gw: 192.168.19.1
I (1284) app: got ip:192.168.19.49

Thanks | Regards, Dipen

[aggarw13]

Namaste @dipen-1337lab, we are happy to help and glad to know that you were able to get the softAP WiFi provisioning application running on your device 🙂.

As you have been able to resolve your question, I will go ahead and resolve the issue. If you have any further questions or concerns, feel free to re-open the issue.

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

I'm running into the below error after successfully completing the SoftAP provisioning. E (61777) WIFI: WIFI_On: Failed to init event loop -1

The logs file is enclosed. WIFI_On Failed to init event loop -1.txt

Debugging reveals that the esp_event_loop_init() gets called twice. First when carrying out SoftAP provisioning and then, as part of the AWS IoT Network Manager initialization, when running our custom application.

The program flow is as below. This flow is the same as that for any Amazon FreeRTOS demo. For SoftAP provisioning, the relevant code in the esp_prov_example.c source file is ported in the main.c source file.

• app_main();
• prvMiscInitialization();
• SYSTEM_Init();
• softAP_provisioning(); --> A new function created in main.c file, which run the SoftAP provisioning
• vDevModeKeyProvisioning();
• DEMO_RUNNER_RunDemos();

From the AFR 202002.00 FreeRTOS project folder perspective, could you help me with an appropriate placeholder (i.e., location in specific source file) for carrying out SoftAP provisioning?

Also, the event_handler() function as available in the esp_prov_example.c file and in the iot_wifi.c file has differences. My understanding is that there should be a single Event handler with all the cases merged. Is my understanding correct?

PS. I am unable to re-open this issue. It's okay as long as you guys continue providing me with the support ;)

Thanks | Regards, Dipen

[aggarw13]

Namaste @dipen-1337lab, with your changes of supporting softAP Provisioning within the main.c file, it makes sense that the WiFi_On function is being called twice, and thus, the Network Manager's functionality of initializing WiFi will not be required for your purpose if your softAP_provisioning function has responsibility of initializing the WiFi connection with an access point after it has already been provisioned with the credentials.

For sanity checking, can you verify that the issue gets resolved in your application after removing the call to WiFi_On in the Network Manager and hard-coding the network type as AWSIOT_NETWORK_TYPE_WIFI here? This avoids the repeated call to WiFi_On and thereby, to esp_event_loop_init. Here is the patch for the changes: patch.txt

From the AFR 202002.00 FreeRTOS project folder perspective, could you help me with an appropriate placeholder (i.e., location in specific source file) for carrying out SoftAP provisioning?

If the above changes work for your custom application, I recommend that you refactor the Network Manager to support the softAP Provisioning functionality i.e. make calls to softAP manager functions if device isn't provisioned with credentials. As you can notice that the aws_iot_network_manager.c file contains code blocks for BLE based WiFi Provisioning, you can refer to that and extend this file to with logic for softAP provisioning.

A starting point would be to add the if-else logic of either entering SoftAP mode (when credentials are not present in device) or connecting to access point (when device is already provisioned) in the _wifiEnable function within the file.

As you update the network manager, it would be important to ensure that the esp_event_loop_init is called only once. Depending on whether your application calls WIFI_On after the device receives credentials through softAP Provisioning, you may consider removing the call to esp_event_loop_init here in the implementation of WIFI_On.

Also, the event_handler() function as available in the esp_prov_example.c file and in the iot_wifi.c file has differences. My understanding is that there should be a single Event handler with all the cases merged. Is my understanding correct?

That is a good point considering that the event handler can be registered only once through the esp_event_loop_init function. @shubhamkulkarni97, can help to clarify whether there needs to be a merger of the event handlers of iot_wifi.c and esp_prov_example.c?

[shubhamkulkarni97]

Namaste @dipen-1337lab @aggarw13,

Yes, single event handler should be used and must have all the required cases covered. You can merge the cases required for softAP provisioning in event handler declared in iot_wifi.c

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

Thank you for re-opening this issue and, your continued assistance.

For sanity checking, can you verify that the issue gets resolved in your application after removing the call to WiFi_On in the Network Manager and hard-coding the network type as AWSIOT_NETWORK_TYPE_WIFI here? This avoids the repeated call to WiFi_On and thereby, to esp_event_loop_init. Here is the patch for the changes: patch.txt

So, after applying the suggested changes as per the patch, an abort() results.

../freertos_kernel/queue.c:1429 (xQueueSemaphoreTake)- assert failed!
abort() was called at PC 0x4008fdd3 on core 0

Below is the console log.

I (61519) main: SoftAP stopped
I (61529) app_prov: Provisioning stopped
I (71319) PKCS11: Initializing NVS partition: "storage"
1 7123 [main] Write certificate...
2 7137 [iot_thread] [INFO ][DEMO][71370] ---------STARTING DEMO---------

I (71459) BTDM_INIT: BT controller compile version [9891bc6]
3 7137 [iot_thread] [INFO ][INIT][71370] SDK successfully initialized.
I (71469) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
GAP procedure initiated: stop advertising.
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=0 adv_filter_policy=0 adv_itvl_min=480 adv_itvl_max=960
../freertos_kernel/queue.c:1429 (xQueueSemaphoreTake)- assert failed!
abort() was called at PC 0x4008fdd3 on core 0
0x4008fdd3: xQueueSemaphoreTake at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../freertos_kernel/queue.c:2368

ELF file SHA256: bcfd4027b4273ba62e6067890762fac52bd6dd3340a0c6ecc9cef164fc1dc5a4

Backtrace: 0x40093264:0x3ffe1a00 0x40093491:0x3ffe1a20 0x4008fdd3:0x3ffe1a40 0x400d6bed:0x3ffe1a80 0x400d3503:0x3ffe1b30 0x400d352f:0x3ffe1b50 0x400d37e7:0x3ffe1b70 0x400d39e6:0x3ffe1b90 0x400d3a5f:0x3ffe1bb0 0x4015ee93:0x3ffe1bd0
0x40093264: invoke_abort at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../vendors/espressif/esp-idf/components/esp32/panic.c:715

0x40093491: abort at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../vendors/espressif/esp-idf/components/esp32/panic.c:715

0x4008fdd3: xQueueSemaphoreTake at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../freertos_kernel/queue.c:2368

0x400d6bed: WIFI_ConnectAP at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c:1369

0x400d3503: _wifiConnectAccessPoint at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../firmware/network_manager/aws_iot_network_manager.c:926

0x400d352f: _wifiEnable at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../firmware/network_manager/aws_iot_network_manager.c:926

0x400d37e7: AwsIotNetworkManager_EnableNetwork at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../firmware/network_manager/aws_iot_network_manager.c:926

0x400d39e6: _initialize at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../firmware/demo_runner/iot_demo_freertos.c:390

0x400d3a5f: runDemoTask at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../firmware/demo_runner/iot_demo_freertos.c:390

0x4015ee93: _threadRoutineWrapper at D:\gitHub\dipen-1337lab\repositories\plugpoint-evcs-firmware\dev\plugpoint-evcs-firmware\evcs-p2\FreeRTOS\build/../libraries/abstractions/platform/freertos/iot_threads_freertos.c:318

Rebooting...

If the above changes work for your custom application ...

A starting point would be to add the if-else logic ...

As you update the network manager ...

I shall keep all these points in mind when proceeding further. Thank you!

Yes, single event handler should be used and must have all the required cases covered. You can merge the cases required for softAP provisioning in event handler declared in iot_wifi.c

Noted. Thank you!

Thanks | Regards, Dipen

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

Here's another approach I have been attempting. Haven't been able to resolve the issue but would like to understand the viability of the same.

So, I found this esp_event_loop_deinit() API.

The corresponding C code in the esp-idf\components\esp_event\event_loop_legacy.c is as below. Unfortunately, I'm unable to find this file in the ESP-IDF or amazon-freertos repositories but happened to find it the an earlier amazon-freertos repository downloaded locally quite some time ago.

esp_err_t esp_event_loop_deinit(void) 
{
    if (!s_initialized) {
        ESP_LOGE(TAG, "system event loop not initialized");
        return ESP_ERR_INVALID_STATE;
    }

    esp_err_t err = esp_event_handler_unregister(SYSTEM_EVENT, ESP_EVENT_ANY_ID, esp_event_post_to_user);
    if (err != ESP_OK) {
        return err;
    }

    err = esp_event_loop_delete_default();
    if (err != ESP_OK && err != ESP_ERR_INVALID_STATE) {
        return err;
    }

    s_initialized = false;
    s_event_handler_cb = NULL;
    s_event_ctx = NULL;
    return ESP_OK;
}

On above lines, I created a new esp_event_loop_deinit() function in AFR 202002.00, with the below source code in the event_loop.c file.

void esp_event_loop_deinit( void )
{
    s_event_init_flag = false;
    s_event_handler_cb = NULL;
    s_event_ctx = NULL;
    s_event_queue = NULL;

    esp_wifi_disconnect();
}

This function is called within the stop_prov_task(), right at the end, just before vTaskDelete(NULL);

This somewhat works with the below errors. My application does proceed for execution. (Error 1) E (75424) event: Event loop not initialized via esp_event_loop_init, but esp_event_send called

(Error 2) I (83754) WIFI: SYSTEM_EVENT_STA_DISCONNECTED: 201 followed by I (84994) WIFI: SYSTEM_EVENT_STA_CONNECTED and I (85564) WIFI: SYSTEM_EVENT_STA_GOT_IP

Partial detailed log is as below.

I (75404) main: SoftAP stopped
I (75404) app_prov: Provisioning stopped
I (75424) wifi: state: run -> init (0)
I (75424) wifi: pm stop, total sleep time: 23064205 us / 31788379 us

I (75424) wifi: new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
E (75424) event: Event loop not initialized via esp_event_loop_init, but esp_event_send called
I (81274) PKCS11: Initializing NVS partition: "storage"
1 8122 [main] Write certificate...
2 8136 [iot_thread] [INFO ][DEMO][81360] ---------STARTING DEMO---------

I (81414) BTDM_INIT: BT controller compile version [9891bc6]
3 8136 [iot_thread] [INFO ][INIT][81360] SDK successfully initialized.
I (81424) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
GAP procedure initiated: stop advertising.
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=0 adv_filter_policy=0 adv_itvl_min=480 adv_itvl_max=960
I (83754) WIFI: SYSTEM_EVENT_STA_DISCONNECTED: 201
I (84874) wifi: new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (84874) wifi: state: init -> auth (b0)
I (84874) wifi: state: auth -> assoc (0)
I (84874) wifi: state: assoc -> run (10)
I (84994) wifi: connected with tkkfnBYMlkv6TeKM2, channel 1, BW20, bssid = 14:ae:85:e6:45:86
I (84994) wifi: pm start, type: 1

I (84994) WIFI: SYSTEM_EVENT_STA_CONNECTED
I (85564) event: sta ip: 192.168.29.49, mask: 255.255.255.0, gw: 192.168.29.1
I (85564) WIFI: SYSTEM_EVENT_STA_GOT_IP
I (85564) dual_outlet: INFO : Entering a short (20 seconds) delay for the MQTT CONNECT operation to complete before other threads are started.
4 8551 [iot_thread] [INFO ][DEMO][85510] Successfully initialized the demo. Network type for the demo: 1
I (85574) dual_outlet: INFO : Thread pvEsp32GetTimeOverSntp() started.
I (85594) dual_outlet: Time is not set yet. Attempting to get the current time over NTP.
I (85604) dual_outlet: Waiting for system time to be set... (1/10)
I (85574) dual_outlet: INFO : Thread pvOtaUpdateTask() started.
I (85574) dual_outlet: INFO : Thread pvMqttTask() started. Delaying task...
5 8556 [iot_thread] [INFO ][MQTT][85560] MQTT library successfully initialized.
6 8556 [iot_thread] [INFO ][DEMO][85560] OTA demo version 0.9.2

7 8556 [iot_thread] [INFO ][DEMO][85560] Connecting to broker...

Further, on restarting the device, the device does not connect to the Access Point. Console log is as below.

0 1 [main] Initializing lwIP TCP stack
I (78) wifi: wifi driver task: 3ffd5e38, prio:23, stack:3584, core=0
I (78) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (78) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (98) wifi: wifi firmware version: c94f3e6
I (98) wifi: config NVS flash: enabled
I (98) wifi: config nano formating: disabled
I (98) wifi: Init dynamic tx buffer num: 32
I (98) wifi: Init data frame dynamic rx buffer num: 32
I (108) wifi: Init management frame dynamic rx buffer num: 32
I (108) wifi: Init management short buffer num: 32
I (118) wifi: Init static rx buffer size: 1600
I (118) wifi: Init static rx buffer num: 10
I (128) wifi: Init dynamic rx buffer num: 32
I (128) app_prov: Found ssid tkkfnBYMlkv6TeKM2
I (128) app_prov: Found password 04Jun#2020
I (128) main: Starting WiFi station
I (228) phy: phy_version: 4102, 2fa7a43, Jul 15 2019, 13:06:06, 0, 0
I (228) wifi: mode : sta (4c:11:ae:df:af:9c)
I (228) PKCS11: Initializing NVS partition: "storage"
E (228) PKCS11: failed nvs get file size 4354 0
1 17 [main] Write certificate...
I (388) wifi: new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (388) wifi: state: init -> auth (b0)
2 33 [iot_thread] [INFO ][DEMO][330] ---------STARTING DEMO---------

I (398) BTDM_INIT: BT controller compile version [9891bc6]
3 34 [iot_thread] [INFO ][INIT][340] SDK successfully initialized.
I (408) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (448) wifi: state: auth -> assoc (0)
I (488) wifi: state: assoc -> run (10)
I (588) wifi: connected with tkkfnBYMlkv6TeKM2, channel 1, BW20, bssid = 14:ae:85:e6:45:86
I (588) wifi: pm start, type: 1

GAP procedure initiated: stop advertising.
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=0 adv_filter_policy=0 adv_itvl_min=480 adv_itvl_max=960
E (718) WIFI: WIFI_On: Failed to init event loop -1
4 67 [iot_thread] [ERROR][DEMO][670] Failed to intialize all the networks configured for the device.
5 67 [iot_thread] [INFO ][INIT][670] SDK cleanup done.
6 67 [iot_thread] [ERROR][DEMO][670] Failed to initialize the demo. exiting...
I (7828) event: sta ip: 192.168.29.49, mask: 255.255.255.0, gw: 192.168.29.1
I (7828) main: got ip:192.168.29.49
... indefinite wait ...

Sorry, I know all this is getting complicated but the point to make here is, is there a way of isolating the softAP provisioning task and then, running the application afresh with Wi-Fi initialization as it normally happens without the softAP provisioning task?

Thanks | Regards, Dipen

[aggarw13]

Namaste @dipen-1337lab,

Thanks for testing with the patch and sharing the call-stack of the abort failure. The cause of the abort failure is from the missed initialization of the xWiFiSem semaphore which occurs in the WIFI_On function (as can be seen here). The patch I provided removes the call to WIFI_On as the intention of the changes in the patch was to test whether removing the second call to esp_event_loop_init avoided the original error (of E (61777) WIFI: WIFI_On: Failed to init event loop -1) you had encountered .

If you were to test the patch's intention again, you could remove the call to esp_event_loop_init from within the WIFI_On function (at this location) and re-instate the call to WIFI_On from the Network Manager (at this location). If this test works, then you will be able to use either of the architecture approaches (of isolating the softAP provisioning functionality in a separate task OR updating the network manager to support softAP Provisioning) as long as the underlying esp_event_loop_init function is called only once.

Regarding your alternative approach of adding an esp_event_loop_deinit function, I will request further input from @shubhamkulkarni97 as that relates to the initializing the ESP-IDF's event loop once for the softAP provisioning task and then re-initializing it for the main application's routine (that occurs post device provisioning).
My thought is that as the event loop within the ESP-IDF SDK that ships with 202002.00 does not expose a esp_event_loop_deinit function, it may be risky to add implementation for it unless @shubhamkulkarni97 can provide a patch for it.

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

If you were to test the patch's intention again, you could remove the call to esp_event_loop_init from within the WIFI_On function (at this location) and re-instate the call to WIFI_On from the Network Manager (at this location).

Below is the diff for testing your suggested changes.

$ git diff evcs-p2/FreeRTOS/firmware/demo_runner/iot_demo_freertos.c
diff --git a/evcs-p2/FreeRTOS/firmware/demo_runner/iot_demo_freertos.c b/evcs-p2/FreeRTOS/firmware/demo_runner/iot_demo_freertos.c
index 2a81aaa..1a6616a 100644
--- a/evcs-p2/FreeRTOS/firmware/demo_runner/iot_demo_freertos.c
+++ b/evcs-p2/FreeRTOS/firmware/demo_runner/iot_demo_freertos.c
@@ -60,7 +60,8 @@ IotNetworkManagerSubscription_t subscription = IOT_NETWORK_MANAGER_SUBSCRIPTION_
 IotSemaphore_t demoNetworkSemaphore;

 /* Variable used to indicate the connected network. */
-static uint32_t demoConnectedNetwork = AWSIOT_NETWORK_TYPE_NONE;
+//static uint32_t demoConnectedNetwork = AWSIOT_NETWORK_TYPE_NONE;
+static uint32_t demoConnectedNetwork = AWSIOT_NETWORK_TYPE_WIFI;

 #if defined( MQTT_DEMO_TYPE_ENABLED )
     #if BLE_ENABLED

$ git diff evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c
diff --git a/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c b/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c
index bb901e8..40b03b0 100644
--- a/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c
+++ b/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/ports/wifi/iot_wifi.c
@@ -275,11 +275,12 @@ WIFIReturnCode_t WIFI_On( void )
         return eWiFiSuccess;
     }

-    esp_err_t ret = esp_event_loop_init(event_handler, NULL);
-    if (ret != ESP_OK) {
-        ESP_LOGE(TAG, "%s: Failed to init event loop %d", __func__, ret);
-        goto err;
-    }
+    esp_err_t ret;
+//    esp_err_t ret = esp_event_loop_init(event_handler, NULL);
+//    if (ret != ESP_OK) {
+//        ESP_LOGE(TAG, "%s: Failed to init event loop %d", __func__, ret);
+//        goto err;
+//    }

     wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
     ret = esp_wifi_init(&cfg);

The corresponding Console Log is as below. The suggested change doesn't result in an abort failure but instead, an indefinite wait.

I (67516) main: SoftAP stopped
I (67516) app_prov: Provisioning stopped
I (71306) PKCS11: Initializing NVS partition: "storage"
1 7122 [main] Write certificate...
2 7135 [iot_thread] [INFO ][DEMO][71350] ---------STARTING DEMO---------

3 7135 [iot_thread] [INFO ][INIT][71350] SDK successfully initialized.
I (71446) BTDM_INIT: BT controller compile version [9891bc6]
I (71446) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
GAP procedure initiated: stop advertising.
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=0 adv_filter_policy=0 adv_itvl_min=480 adv_itvl_max=960
... indefinite wait ...

A starting point would be to add the if-else logic of either entering SoftAP mode (when credentials are not present in device) or connecting to access point (when device is already provisioned) in the _wifiEnable function within the file.

I shall now attempt this suggested if-else logic within the _wifiEnable() function to integrate the SoftAP provisioning in the base code.

Thanks | Regards, Dipen

[dipen-1337lab]

Namaste @shubhamkulkarni97,

Regarding your alternative approach of adding an esp_event_loop_deinit function, I will request further input from @shubhamkulkarni97 as that relates to the initializing the ESP-IDF's event loop once for the softAP provisioning task and then re-initializing it for the main application's routine (that occurs post device provisioning). My thought is that as the event loop within the ESP-IDF SDK that ships with 202002.00 does not expose a esp_event_loop_deinit function, it may be risky to add implementation for it unless @shubhamkulkarni97 can provide a patch for it.

Requesting your assistance on the alternate approach I am attempting, but haven't been successful. Could you confirm if this is feasible as part of the implementation?

Below is the diff and console log for my attempt to isolate the softAP provisioning and then, run the AFR application afresh.

Diff for app_prov.c file and event_loop.c file

$ git diff evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/proj04/application_code/app_prov.c
diff --git a/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/proj04/application_code/app_prov.c b/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/proj04/application_code/app_prov.c
index c1d518d..30ee4d6 100644
--- a/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/proj04/application_code/app_prov.c
+++ b/evcs-p2/FreeRTOS/vendors/espressif/boards/esp32/proj04/application_code/app_prov.c
@@ -123,6 +125,11 @@ static void stop_prov_task(void * arg)

     ESP_LOGI(TAG, "Provisioning stopped");

+    esp_event_loop_deinit();
+    esp_wifi_stop();
+    esp_wifi_deinit();
+
     vTaskDelete(NULL);
 }

$ git diff evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c
warning: LF will be replaced by CRLF in evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c.
The file will have its original line endings in your working directory
diff --git a/evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c b/evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c
index 9dbefb4..d78eb07 100644
--- a/evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c
+++ b/evcs-p2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/event_loop.c
@@ -38,6 +38,8 @@ static QueueHandle_t s_event_queue = NULL;
 static system_event_cb_t s_event_handler_cb = NULL;
 static void *s_event_ctx = NULL;

+static TaskHandle_t xEspEventLoopTaskHandle = NULL;
+
 static esp_err_t esp_event_post_to_user(system_event_t *event)
 {
     if (s_event_handler_cb) {
@@ -117,10 +119,23 @@ esp_err_t esp_event_loop_init(system_event_cb_t cb, void *ctx)
     s_event_ctx = ctx;
     s_event_queue = xQueueCreate(CONFIG_SYSTEM_EVENT_QUEUE_SIZE, sizeof(system_event_t));

+//    xTaskCreatePinnedToCore(esp_event_loop_task, "eventTask",
+//            ESP_TASKD_EVENT_STACK, NULL, ESP_TASKD_EVENT_PRIO, NULL, 0);
     xTaskCreatePinnedToCore(esp_event_loop_task, "eventTask",
-            ESP_TASKD_EVENT_STACK, NULL, ESP_TASKD_EVENT_PRIO, NULL, 0);
+            ESP_TASKD_EVENT_STACK, NULL, ESP_TASKD_EVENT_PRIO, xEspEventLoopTaskHandle, 0);

     s_event_init_flag = true;
     return ESP_OK;
 }

+
+void esp_event_loop_deinit( void )
+{
+    s_event_init_flag = false;
+    s_event_handler_cb = NULL;
+    s_event_ctx = NULL;
+    s_event_queue = NULL;
+
+    if( NULL != xEspEventLoopTaskHandle )
+        vTaskDelete( xEspEventLoopTaskHandle );
+}

Associated Console Log (for above changes)

I (64709) main: SoftAP stopped
I (64709) app_prov: Provisioning stopped
I (64719) wifi: state: run -> init (0)
I (64719) wifi: pm stop, total sleep time: 23542477 us / 30753503 us

I (64729) wifi: new:<11,0>, old:<11,0>, ap:<255,255>, sta:<11,0>, prof:11
E (64729) event: Event loop not initialized via esp_event_loop_init, but esp_event_send called
E (64739) event: Event loop not initialized via esp_event_loop_init, but esp_event_send called
I (64769) wifi: flush txq
I (64769) wifi: stop sw txq
I (64769) wifi: lmac stop hw txq
I (64789) wifi: Deinit lldesc rx mblock:10
I (71319) PKCS11: Initializing NVS partition: "storage"
1 7123 [main] Write certificate...
2 7136 [iot_thread] [INFO ][DEMO][71360] ---------STARTING DEMO---------

I (71449) BTDM_INIT: BT controller compile version [9891bc6]
3 7136 [iot_thread] [INFO ][INIT][71360] SDK successfully initialized.
I (71459) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
GAP procedure initiated: stop advertising.
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=0 adv_filter_policy=0 adv_itvl_min=480 adv_itvl_max=960
I (71759) wifi: wifi driver task: 3ffdd7e0, prio:23, stack:3584, core=0
I (71759) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (71759) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (71779) wifi: wifi firmware version: c94f3e6
I (71779) wifi: config NVS flash: enabled
I (71779) wifi: config nano formating: disabled
I (71779) wifi: Init dynamic tx buffer num: 32
I (71789) wifi: Init data frame dynamic rx buffer num: 32
I (71789) wifi: Init management frame dynamic rx buffer num: 32
I (71799) wifi: Init management short buffer num: 32
I (71799) wifi: Init static rx buffer size: 1600
I (71809) wifi: Init static rx buffer num: 10
I (71809) wifi: Init dynamic rx buffer num: 32
E (71819) wifi: esp_wifi_connect 1145 wifi not start

Thanks | Regards, Dipen

[shubhamkulkarni97]

@dipen-1337lab, Sorry for late response.

Is it possible for you to use master branch for your application? master branch supports latest version of ESP-IDF (v4.2) and also supports softAP provisioning demo.

This is because the changes suggested earlier should be tested before using in application/production.

[dipen-1337lab]

Namaste @shubhamkulkarni97 and @aggarw13,

Is it possible for you to use master branch for your application?

Unfortunately, this would not be possible at this point in time, with our product's deliverable in the near future. Our firmware is developed around AFR 202002.00 and, is stable to a great extent. Porting our application to the master branch could potentially result in quite a lot of rework (fix all the breaks) and/ or, modifications specific to the latest AFR base code. CMake in itself is quite complex.

Having said that, we would surely have to periodically update our firmware in line to the continuous AFR releases and, to leverage the new features being made available for the customers.

A starting point would be to add the if-else logic of either entering SoftAP mode (when credentials are not present in device) or connecting to access point (when device is already provisioned) in the _wifiEnable function within the file.

With regards to the update on the original issue, I'm happy to share another interim update, that I have managed to integrate the softAP provisioning in our application, with the above suggested approach. So once again, I'm grateful to both of you guys for your continuous assistance.

The next issue, and hopefully - the last one, is that the application always enters the softAP provisioning mode (after Soft Reset as well as Hard Reset) even after the Wi-Fi credentials of the AP have been provided once. This, to my understanding seems because of, either the Wi-Fi credentials not getting correctly saved in the Flash or, not getting correctly populated from the Flash, on reset. I'm currently debugging the same.

Thanks | Regards, Dipen

[shubhamkulkarni97]

@dipen-1337lab Good to know that you were able to successfully integrate softAP provisioning.

WiFi credentials are not retrieved after reset because we set WiFi storage to RAM, please check the call here. This is because storing and retrieving credentials from flash is handled by provisioning framework in AFR. You can try removing the call in iot_wifi.c.

Here are some points that need to be considered even if the softAP provisioning works with these changes:

• Using esp-idf provisioning framework will store the credentials in default nvs partition named nvs. For production scenarios, it is recommended to encrypt this nvs partition.
• BLE provisioning is supported in AFR provisioning framework (in AFR 202002.00 as well) . Did you consider using BLE provisioning in your application? Is there any specific reason for opting softAP provisioning?

softAP provisioning was integrated with AFR provisioning framework in commit https://github.com/aws/amazon-freertos/commit/de393071e7e0ebcdb8c4201a04e2919696209d4d @dachalco Is it possible to cherry-pick this commit on top of AFR 202002.00? Are there any dependencies due to which this commit will not work on specified release?

Regards, Shubham

[dipen-1337lab]

Namaste @aggarw13 and @shubhamkulkarni97,

WiFi credentials are not retrieved after reset because we set WiFi storage to RAM, please check the call here. This is because storing and retrieving credentials from flash is handled by provisioning framework in AFR. You can try removing the call in iot_wifi.c.

Thank you for this pointer. After commenting the suggested esp_wifi_set_storage() call, the execution was halting with an error wifi not start. This was resolved by creating a new function WIFI_StartSTA() in the iot_wifi.c source file.

Further, it is observed that the softAP provisioning works as desired with both the changes - by commenting the esp_wifi_set_storage() call as well as by calling esp_wifi_set_storage(WIFI_STORAGE_FLASH) i.e., passing WIFI_STORAGE_FLASH instead of WIFI_STORAGE_RAM to the esp_wifi_set_storage function. Just sharing this observation for your reference.

And finally, softAP provisioning for configuring the Wi-Fi credentials within the device is working as desired. Phew! ;-)

• The git status log for the files modified is shared here - log_git_status.diff.txt.
• The git diff log for the corresponding code changes is shared here - log_git_diff.diff.txt

Kindly remove the .txt extension for viewing the diff in the above files. Please review the code changes, especially the modification within the _wifiEnable function and, the newly created WIFI_StartSTA function. A new #define IOT_ENABLE_WIFI_SOFTAP_PROVISIONING has been created in the iot_config.h header file.

The function call hierarchy is implemented as: _wifiEnable --> WIFI_On --> softAP_provisioning --> connect to AP with available credentials or, carry out Wi-Fi provisioning. Do share your review comments!

• Using esp-idf provisioning framework will store the credentials in default nvs partition named nvs. For production scenarios, it is recommended to encrypt this nvs partition.

Noted. I shall explore this for the implementation. Some additional work to be done includes,

1. save multiple Wi-Fi credentials in the NVS and, then iterate the connection to the different APs in case of Wi-Fi failure.
2. support changing of the password and, connect with the new password at next device power ON.

• BLE provisioning is supported in AFR provisioning framework (in AFR 202002.00 as well) . Did you consider using BLE provisioning in your application? Is there any specific reason for opting softAP provisioning?

Yes, I'm aware of the BLE provisioning supported in AFR 202002.00. softAP provisioning is preferred to avoid BLE, which to our understanding is somewhat memory heavy. Also, there's a dependency on the AWS Cognito.

softAP provisioning was integrated with AFR provisioning framework in commit de39307

Thank you for this reference!

With that, I'd like to close this issue. Truly grateful for your continuous assistance over the last three weeks. Wishing you continued success!

Thanks | Regards, Dipen

[aggarw13]

Namaste @dipen-1337lab, I am glad that you have been able to achieve desired WiFi provisioning functionality. The changes to the network manager within diff file you have shared look good to me.

Thank you @shubhamkulkarni97 for providing guidance on the issue.

[dachalco]

@shubhamkulkarni97

@dachalco Is it possible to cherry-pick this commit on top of AFR 202002.00? Are there any dependencies due to which this commit will not work on specified release?

With additional build logic (cmake, etc.) should be able to work with 202002.00 since that release uses an IDF version based on v3.3 (IIRC). There are dependency collisions between our coreHTTP and IDF component esp_http_server. Their HTTP parsers, having slightly different incompatible versions, need to be coordinated at link time.