ESPNOW stops receiving when ESP32-S3 in STA mode connects to WiFi AP (AEGHB-94)

[ramiws]

I created a minimalist code to show the issue. Yes I disabled power saving and it is not about setting the channel on the sender. I took the new channel that was set by the AP of my router and put it in the sender. Then the espnow stops arriving once the esp is connected to AP. I even made a sender that keeps sending over all WiFi channels. You can use your sender example and then my code on the receiver to witness the issue.

IDF 5.0 and 5.0.1

/*
   This example shows how to use ESPNOW.
   Prepare two device, one for sending ESPNOW data and another for receiving
   ESPNOW data.
*/
#include <stdlib.h>
#include "nvs_flash.h"
#include "esp_random.h"
#include "esp_event.h"
#include "esp_netif.h"
#include "esp_wifi.h"
#include "esp_log.h"
#include "esp_mac.h"
#include "esp_now.h"

static const char *TAG = "espnow_example";

static void wifi_event_handler(void* arg, esp_event_base_t event_base,int32_t event_id, void* event_data){
    if (event_base == WIFI_EVENT && event_id == WIFI_EVENT_STA_START) {
      esp_wifi_connect(); // <------ 2- comment this line to get ESPNOW to work, uncomment to break ESPNOW. ( Sender is using the new channel yes)
    }
}
static void example_wifi_init(void)
{
    ESP_ERROR_CHECK(esp_netif_init());
    ESP_ERROR_CHECK(esp_event_loop_create_default());
    wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT();
    ESP_ERROR_CHECK( esp_wifi_init(&cfg) );
    ESP_ERROR_CHECK(esp_event_handler_instance_register(WIFI_EVENT,ESP_EVENT_ANY_ID, &wifi_event_handler, NULL, NULL));                                                
    ESP_ERROR_CHECK( esp_wifi_set_storage(WIFI_STORAGE_RAM) );
    esp_wifi_set_ps(WIFI_PS_NONE);// <------ it is set 
    ESP_ERROR_CHECK( esp_wifi_set_mode(WIFI_MODE_STA) );

    wifi_config_t wifi_config = {
      .sta = {
          .ssid = "", // <------ 1- set your AP so esp can connect to 
            .password = "",
          .sae_pwe_h2e = WPA3_SAE_PWE_UNSPECIFIED,
      },
    };

    ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
    ESP_ERROR_CHECK( esp_wifi_start());

}

static void example_espnow_send_cb(const uint8_t *mac_addr, esp_now_send_status_t status)
{

}

static void example_espnow_recv_cb(const esp_now_recv_info_t *info, const uint8_t *data, int len)
{
    ESP_LOGE(TAG, "Got ESPNOW msg");

}

static esp_err_t example_espnow_init(void)
{
    ESP_ERROR_CHECK( esp_now_init() );
    ESP_ERROR_CHECK( esp_now_register_send_cb(example_espnow_send_cb) );
    ESP_ERROR_CHECK( esp_now_register_recv_cb(example_espnow_recv_cb) );
    /* Set primary master key. Disable in sender for simplicity, even though it does not matter for this bug */
    //ESP_ERROR_CHECK( esp_now_set_pmk((uint8_t *)CONFIG_ESPNOW_PMK) ); 

    return ESP_OK;
}

void app_main(void)
{
    esp_err_t ret = nvs_flash_init();
    if (ret == ESP_ERR_NVS_NO_FREE_PAGES || ret == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        ESP_ERROR_CHECK( nvs_flash_erase() );
        ret = nvs_flash_init();
    }
    ESP_ERROR_CHECK( ret );

    example_wifi_init();
    example_espnow_init();

}

[lhespress]

@ramiws Did you check the receiver's channel when it connected with the AP, Is it the same with the sender?

[ramiws]

@ramiws Did you check the receiver's channel when it connected with the AP, Is it the same with the sender?

Yes of course. I tried the following: 1- I look at what channel the receiver is now on after connecting to the AP, I copy it to the sender. Nothing comes. 2- I made a sender that try over all the WiFi channels so no matter what receiver is on, something should arrive. It works and it does arrive, until connected to WiFi. It stops. 3- I forced the AP to be on channel 1 and everything on channel 1. Messages come normally, till it connects to WiFi and still the channel is 1.

So please try code on IDF 5.0 or 5.0.1 Use this for sender and receiver https://github.com/espressif/esp-idf/tree/v5.0.1/examples/wifi/espnow On receiver paste my code. @lhespress Can you confirm bug?

[lhespress]

@ramiws I checkout your code use ESP32-S3 on 5.0.1, it works fine:

Receiver on ESP32-S3 LOG:

I (634) wifi:mode : sta (84:f7:03:80:09:50)
I (634) wifi:enable tsf
I (634) ESPNOW: espnow [version: 1.0] init
I (704) wifi:new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (1664) wifi:state: init -> auth (b0)
E (1674) espnow_example: Got ESPNOW msg
I (1714) wifi:state: auth -> assoc (0)
I (1754) wifi:state: assoc -> run (10)
E (1774) espnow_example: Got ESPNOW msg
I (1994) wifi:connected with Redmi_9BDB_2.4G, aid = 1, channel 1, BW20, bssid = 5c:02:14:71:d3:93
I (1994) wifi:security: WPA2-PSK, phy: bgn, rssi: -34
I (2004) wifi:pm start, type: 0

I (2004) wifi:set rx beacon pti, rx_bcn_pti: 0, bcn_timeout: 0, mt_pti: 25000, mt_time: 10000
I (2174) wifi:AP's beacon interval = 102400 us, DTIM period = 1
E (2814) espnow_example: Got ESPNOW msg
E (3844) espnow_example: Got ESPNOW msg
E (4834) espnow_example: Got ESPNOW msg
E (5874) espnow_example: Got ESPNOW msg
E (6914) espnow_example: Got ESPNOW msg
E (7914) espnow_example: Got ESPNOW msg
E (8914) espnow_example: Got ESPNOW msg
E (9914) espnow_example: Got ESPNOW msg
E (10924) espnow_example: Got ESPNOW msg

Sender on ESP32 LOG:

I (615) wifi_init: rx ba win: 6
I (615) wifi_init: tcpip mbox: 32
I (625) wifi_init: udp mbox: 6
I (625) wifi_init: tcp mbox: 6
I (625) wifi_init: tcp tx win: 5744
I (635) wifi_init: tcp rx win: 5744
I (635) wifi_init: tcp mss: 1440
I (645) wifi_init: WiFi IRAM OP enabled
I (645) wifi_init: WiFi RX IRAM OP enabled
I (655) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07
I (775) wifi:mode : sta (c8:f0:9e:35:93:70)
I (775) wifi:enable tsf
I (775) ESPNOW: espnow [version: 1.0] init
I (5775) espnow_example: Start sending broadcast data
I (6795) espnow_example: send data to ff:ff:ff:ff:ff:ff
I (7795) espnow_example: send data to ff:ff:ff:ff:ff:ff
I (8805) espnow_example: send data to ff:ff:ff:ff:ff:ff
I (9815) espnow_example: send data to ff:ff:ff:ff:ff:ff

you can also replace the firmware and sdkconfig file confirm again: TestBin.zip

[lhespress]

@ramiws If the receiver's wifi channel isn't 1 after connected with the AP, please call

ESP_ERROR_CHECK(esp_wifi_set_channel(CONFIG_ESPNOW_CHANNEL, WIFI_SECOND_CHAN_NONE)); 

before https://github.com/espressif/esp-idf/blob/v5.0.1/examples/wifi/espnow/main/espnow_example_main.c#L320 on the sender.

[ramiws]

@lhespress Thank you very much. I can see the problem is about the mac address used in the sender. When I use FF:FF:FF:FF:FF the messages arrive. But in my case I need to hardcode the mac address of the receiver in the sender. So when I use the STA mac, it does not work. I used the mac that I see when uploading to flash

Connecting.... Chip is ESP32-S3 (revision v0.1) Features: WiFi, BLE Crystal is 40MHz MAC: f4:12:fa:50:39:40 Uploading stub... Running stub...

Is this the correct mac to use?

[lhespress]

@ramiws It's right, did you change it https://github.com/espressif/esp-idf/blob/v5.0.1/examples/wifi/espnow/main/espnow_example_main.c#L39?

[ramiws]

AAAAH, I am losing it. Please be patience. I was trying different senders such as S3 and ESP8266 because I thought it cannot be related to sender chip as it is the same protocol. Now I can pin point that the issue is that when I am using ESP8266 sender the messages arrive if the S3 is not connected to WiFi. But when S3 connects to AP, then it messages do not come. So here is the summary of my testing: Sender S3->S3 works with or without wifi connected. Sender 8266-> S3 wifi not connected ( espnow works) Sender 8266-> S3 wifi connected ( espnow does not work) ( yes I updated the channel on the 8266 sender)

@lhespress Could you please test with esp8266 as sender? I simply use esp_now_send to the receiver mac and wifi_set_channel to set the wifi channel. It works till the S3 connects to wifi. I also tried to kill the hotspot AP from my phone to get the S3 to disconnect and then the espnow messages starts to arrive and work again.

[ramiws]

@lhespress does espressif have commercial support? Are you employee there? I tried to reach out to espressif with no luck.

[lhespress]

@ramiws ra I witness the issue on the ESP8266. The protocol on unicast has a litter difference between ESP8266 and ESP32 . BTW, could you describe the application scenario? The ESP32-C2 will be a better select. Of course, we will also fix it on ESP8266 ASAP.

[ramiws]

@lhespress Okay that is surprising because the espnow packet that esp8266 is sending is arriving. I was expecting the issue is on the esp32 side when it connects to wifi the issue appears. The esp8266 sends out the same packet regardless of esp32 wifi status. It cannot be that I am the first one trying to use espnow with esp8266 and esp32. While having esp32 connected to wifi ap. It is all over the internet that it works, was there some recent change that broke it? I already have 100s of pcbs with esp8266. I cannot simply through them away. C2 are not as widely available. The product is commercial and was using esp8266 to collect data and send it to the esp8266 that is acting as a gateway that collects espnow data and send it to server using wifi/internet. I migrated the gateway from esp8266 to esp32 then I hit this issue.

The production of the commercial product is not stopped due to this issue. Please provide an estimate for the fix. What is the time frame you mean by ASAP. Can you give me a hotpatch binary of espnow that I can use?

[lhespress]

@ramiws Please replace the library as the attachment and check again. espnow.zip

[ramiws]

@lhespress Do I replace both files in the ESP8266 only ?

[lhespress]

@ramiws Right

[ramiws]

@lhespress what framework and version are you using for the esp8266? I am using arduino core for esp8266 which is based on NonOS if I remember correctly I replaced the lib files and got build errors. I guess it is not compatible with the rest of the libraries build_errors.txt

[lhespress]

@ramiws The library is for master branch of ESP8266_RTOS_SDK.

[ramiws]

@lhespress Yes now it works with the RTOS_sdk! Thanks! Could you port the fix to NonOS too? Majority of users still use Arduino/NonOS and plarformio to develop on esp8266. It is also big task for me to port the code to RTOS_sdk, I might face new issues or non-matching APIs. Then there would be more work to power profile the new firmware, as the old one went through that testing already. All those delays in production will have really bad impact on our company.

[AmirHmZz]

Any updates on porting this hotfix to NonOS?

[lhespress]

@AmirHmZz @ramiws The fix is under the internal reviewing, you can replace the library and check again. libnet80211.a.zip

[ramiws]

@AmirHmZz @ramiws The fix is under the internal reviewing, you can replace the library and check again. libnet80211.a.zip

I wasted so much time to rewrite the firmware using rtos esp8266 sdk :( So yea I'm happy to spend more time testing

[ramiws]

@lhespress Yes! it is working with nonos, can I use it for production or should I wait the internal review?

[ramiws]

@lhespress Actually there is packet drop (sendstatus is fail) sometimes. Can you run it for an hour with messages from esp8266 to esp32-s3 every 10 seconds and see how many drops?
I checked the esp32-s3 connection to wifi ap is stable and did not change channel, but still the drops happens.

[lhespress]

@ramiws The fix of NONOS has release on master branch.