search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.35k stars 7.21k forks source link

[Regression][WPA2 Enterprise] WPA2 Enterprise connection fail (IDFGH-5866) #7565

Closed PaulFreund closed 2 years ago

PaulFreund commented 3 years ago

Hello,

our application has been working fine for over a year now. We used ESP IDF tags/v4.1 and now wanted to upgrade to tags/v4.3 (stable). The lowest tested version was tags/v4.1.2 which also was not able to connect to a WPA2 Enterprise network that works with the previous version of ESP IDF. I will try to provide more addditional information but only have limited time available. Is this a known regression?

I already tried the patch from this issue: https://github.com/espressif/esp-idf/issues/7384 and also disabling mbedTLS in WPA supplicant options.

negativekelvin commented 3 years ago

Have you tried latest release/v4.3 branch instead of the tag

PaulFreund commented 3 years ago

I have by now tried: tags/v4.1 - WPA2Enterprise works but I can't use because of pSRAM issues tags/v4.1.1 - WPA2Enterprise Broken tags/v4.1.2 - WPA2Enterprise Broken tags/v4.3 - WPA2Enterprise Broken master - WPA2Enterprise Broken

I will test release/v4.3 as soon as possible

PaulFreund commented 3 years ago

Update:

release/v4.3 - WPA2Enterprise Broken

Little background, I helped debug WPA2E in the past in https://github.com/espressif/esp-idf/issues/2152 and https://github.com/espressif/esp-idf/issues/2324 and it was working without issues since then. Now there are a lot of devices relying on that functionality in a remote location so I have only limited testing capabilities

kapilkedawat commented 3 years ago

Hi @PaulFreund ,

Can you please try on the latest master once? Which patch from https://github.com/espressif/esp-idf/issues/7384 have you tried since there are multiple discussions in that?

Also please provide following:

  1. Radius server details
  2. Auth method used
  3. SDK config
AxelLin commented 3 years ago

I have by now tried: tags/v4.1 - WPA2Enterprise works but I can't use because of pSRAM issues tags/v4.1.1 - WPA2Enterprise Broken

Can you bisect to find bad commit? (This is probably the fastest way to find the problem) The changes between v4.1 and v4.1.1 in wpa_supplicats: $ git shortlog v4.1..v4.1.1 --no-merges components/wpa_supplicant/ Angus Gratton (1): wpa_supplicant: Fix failure to link under some circumstances

GOPTIONS\pfrost (1): Reduce log level of hexdumps to verbose

Hrudaynath Dhabe (2): wpa_supplicant: Fix configurable debug log feature's warning issue wpa_supplicant: Minor bugfix with wpa_supplicant debug logs.

Jiang Jiang Jian (2): Merge branch 'bugfix/clean_wpa2_wifi_deinit' into 'master' Merge branch 'bugfix/wps_fail_reason_code' into 'master'

Nachiket Kukade (5): wpa_supplicant: Allow NULL-padded WPS attributes wpa_supplicant: Fix WPA3 and WPA2 transition related failures wpa_supplicant: Increase PMK Lifetime to a very high value espnow/pmf: Implement ESPNOW + PMF Co-existance esp_wifi: Update wifi lib

kapil.gupta (6): wpa_supplicant: WPS Inter operatability Fixes wpa_supplicant: Support for mbedtls tls handshake wpa_supplicant: Fix invalid pointer deference and memleak wpa_supplicant: Fix IOT issue with latest freeradius ESP-WIFI: Optimize 4way handshake failure time wpa_supplicant: Fix null pointer dereference if eap init failed

ronghulin (1): bugfix: fix softap mode wpa memory leak

PaulFreund commented 3 years ago

Hi @kapilkedawat

I already tried the latest master yesterday evening (CEST) which did not work.

  1. We don't use client certificates and Radius server is part of Cisco infrastructure.
  2. I didn't set it explicitly but in the past it was PEAP with MSCHAPv2
  3. sdkconfig.txt
kapilkedawat commented 3 years ago

Hi @PaulFreund , Just to reconfirm both https://github.com/espressif/esp-idf/commit/e8360fe0756ec592cbd5f4ff4d36946a22561d8f and https://github.com/espressif/esp-idf/issues/7384#issuecomment-913985517 were present in master when you tested this?

Also is it possible to share sniffer capture and logs of the issue? Please note in case mbedTLS is enabled, Please enable logs for both mbedTLS and wpa_supplicant.

AxelLin commented 3 years ago

Hi @PaulFreund , Just to reconfirm both e8360fe and #7384 (comment) were present in master when you tested this?

I think it's not related unless v4.1 includes the fix of https://github.com/espressif/esp-idf/issues/7384#issuecomment-913985517

PaulFreund commented 3 years ago

@kapilkedawat I was definitely on master which included your commit and I think I also tested with the fix enabled but not 100% sure. Unfortuantely I don't have physical access to the devices or can read serial output. I'm getting all log output by intercepting vsprintf and sending it to an InfluxDB.

I'm fine tuning the log output to not overwhelm the network (some devices are also connected via Ethernet) and it looks like the error is an assoc expire. I'll try to get more relevant output.

kapilkedawat commented 3 years ago

@PaulFreund If you are sure about the issue after disabling mbedTLS in tags/v4.1.1.

Can you please try following:

  1. Revert this commit : wpa_supplicant: Fix IOT issue with latest freeradius
  2. Disable mbedTLS in wpa_supplicant

and see if that helps?

Also are you aware of the TLS version cisco infra is using? Again, does the remote setup has any system which can help you to capture the packets over the air?

PaulFreund commented 3 years ago

I just tested with:

  1. 50b4cf92860d99cdd3815793ac29b5e40cdb2323 (wpa_supplicant: Fix IOT issue with latest freeradius)
  2. 59347d6a63c4b587578081778bbb3163c05d6b6e (the commit before that)

I also tried before with tags/v4.3 and mbedTLS disabled but I will retest with that version

We should be able to extract information like the TLS version from the Posts I linked above, the infrastructure has not changed since then, I assume TLSv1 (there should even be some captures IIRC)

Small update: https://github.com/espressif/esp-idf/commit/0c136620d96782984adbbe6b2eded9e58c034f6f is my last tested working commit so far, I'm trying to do a binary search

kapilkedawat commented 3 years ago

I just tested with:

  1. 50b4cf9 (wpa_supplicant: Fix IOT issue with latest freeradius)
  2. 59347d6 (the commit before that)

Both didn't work(MbedTLS was disabled in supplicant config)?

I also tried before with tags/v4.3 and mbedTLS disabled but I will retest with that version

We should be able to extract information like the TLS version from the Posts I linked above, the infrastructure has not changed since then, I assume TLSv1 (there should even be some captures IIRC)

Even if you have older captures please share in case nothing has changed on the server side.

Small update: 0c13662 is my last tested working commit so far, I'm trying to do a binary search

Thanks, we will take a look internally as well.

PaulFreund commented 3 years ago

Sorry yes, both didn't work and CONFIG_WPA_MBEDTLS_CRYPTO is still unset (disabled in menuconfig)

I'm now at 573f5de99ad615364d9acf0f22fc9ee34bacbb09 and it does NOT work so it must be between 0c13662 and this one

PaulFreund commented 3 years ago

Okay I found the offending commit:

77eb201 - WORKING e365d1f - NOT WORKING

Which means e365d1ff602e42758d64c450c928e02e65a7cff5 is the commit that brought the issue. In addition I get

E (36877) wpa: Method private structure allocated failure

with this commit which was not present before

PaulFreund commented 3 years ago

The allocation failure only happens if CONFIG_WPA_MBEDTLS_CRYPTO is unset.

If MBEDTLS in supplicant is enabled it also does NOT work but the allocation failure does not happen.

Just to clarify,

in 77eb201 WPA2 Enterprise works both with MBEDTLS enabled and disabled in e365d1f WPA2 Enterprise does not work both with MBEDTLS enabled and disabled

kapilkedawat commented 3 years ago

@PaulFreund Can you please share partition.csv file? We are trying to check this with the exact config on windows and linux radius servers.

PaulFreund commented 3 years ago 
# Name,   Type, SubType, Offset,  Size, Flags
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x4000,
otadata,  data, ota,     0xd000,  0x2000
phy_init, data, phy,     0xf000,  0x1000,
ota_0,    app,  ota_0,   0x10000, 0x1F0000
ota_1,    app,  ota_1,   0x200000,0x1F0000
# We currently waste 16000 bytes of data because of alignment

Sure :) Thank you for working on it, this is the content

kapilkedawat commented 3 years ago

@PaulFreund Unfortunately this is working for us with hostapd/freeradius. From your linked issues logs, I can see TLS version was used as TLSv1 but since those are very old logs not sure this was due to esp devices or the server itself doesn't support new protocols.

A fresh capture/serial logs will help to determine the issue faster, Is there any chance we can get them?

PaulFreund commented 3 years ago

ESPIDFv4.3_mbedTLS_Enabled.csv

here is a first (anonymised) log of v4.3 with mbedTLS enabled and debug output of wpa and wifi

kapilkedawat commented 3 years ago

@PaulFreund I took a look at the logs, Can you please retry with the latest master? Please make sure https://github.com/espressif/esp-idf/commit/d3a42d787d3d9a0ab916855bb48b1cd530e47961 and patch mentioned in https://github.com/espressif/esp-idf/issues/7384#issuecomment-913985517 both are present? Please capture the logs again.

PaulFreund commented 3 years ago

I will do as soon as possible. Meanwhile, here is the log with mbedTLS disabled in wpa supplicant ESPIDFv4.3_mbedTLS_Disabled.csv

PaulFreund commented 3 years ago

Here is the log of the current master with d3a42d7 included and the fix from the comment. Mbedtls is enabled in supplicant settings ESPIDFv220590d599_mbedTLS_Enabled.csv

kapilkedawat commented 3 years ago

Hi @PaulFreund , do you use make instead of cmake by any chance?

PaulFreund commented 3 years ago

Hi, yes I still have to use make because of some legacy components

kapilkedawat commented 3 years ago

Can you please try this change when mbedTLS disabled? (CONFIG_SHA256 flag is missing in cflags)

diff --git a/components/wpa_supplicant/component.mk b/components/wpa_supplicant/component.mk index c81e71993be..f3fe1a67c82 100644 --- a/components/wpa_supplicant/component.mk +++ b/components/wpa_supplicant/component.mk @@ -35,7 +35,7 @@ ifneq ($(CONFIG_WPA_11KV_SUPPORT), y) src/esp_supplicant/esp_scan.o endif

-CFLAGS += -DCONFIG_DPP -DCONFIG_IEEE80211W -DESP_SUPPLICANT -DIEEE8021X_EAPOL -DEAP_PEER_METHOD -DEAP_TLS -DEAP_TTLS -DEAP_PEAP -DEAP_MSCHAPv2 -DUSE_WPA2_TASK -DCONFIG_WPS2 -DCONFIG_WPS_PIN -DUSE_WPS_TASK -DESPRESSIF_USE -DESP32_WORKAROUND -DCONFIG_ECC -DCONFIG_WNM -Dets -Wno-strict-aliasing +CFLAGS += -DCONFIG_DPP -DCONFIG_IEEE80211W -DESP_SUPPLICANT -DIEEE8021X_EAPOL -DEAP_PEER_METHOD -DEAP_TLS -DEAP_TTLS -DEAP_PEAP -DEAP_MSCHAPv2 -DUSE_WPA2_TASK -DCONFIG_WPS2 -DCONFIG_WPS_PIN -DUSE_WPS_TASK -DESPRESSIF_USE -DESP32_WORKAROUND -DCONFIG_ECC -DCONFIG_WNM -Dets -Wno-strict-aliasing -DCONFIG_SHA256

ifdef CONFIG_ESP32_WIFI_ENABLE_WPA3_SAE CFLAGS += -DCONFIG_WPA3_SAE

PaulFreund commented 3 years ago

Yes that works with current master!

PaulFreund commented 3 years ago

I just tried to compile with mbedtls in current master but I get linking errors 

/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o):(.literal.wpa_sta_disconnected_cb+0x0): undefined reference to `esp_wpa3_free_sae_data'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o):(.literal.esp_supplicant_init+0x48): undefined reference to `esp_wifi_register_wpa3_cb'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o): in function `wpa_sta_disconnected_cb':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/esp_supplicant/src/esp_wpa_main.c:221: undefined reference to `esp_wpa3_free_sae_data'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(esp_wpa_main.o): in function `esp_supplicant_init':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/esp_supplicant/src/esp_wpa_main.c:270: undefined reference to `esp_wifi_register_wpa3_cb'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-prf.o):(.literal.sha1_prf+0x0): undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-prf.o): in function `sha1_prf':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-prf.c:64: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha256-prf.o): in function `sha256_prf_bits':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha256-prf.c:105: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-tlsprf.o): in function `tls_prf_sha1_md5':
/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:95: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:95: undefined reference to `forced_memzero'
/home/User/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/User/ApplicationName/build/wpa_supplicant/libwpa_supplicant.a(sha1-tlsprf.o):/home/User/ApplicationName/esp-idf/components/wpa_supplicant/src/crypto/sha1-tlsprf.c:96: more undefined references to `forced_memzero' follow
collect2: error: ld returned 1 exit status
make: *** [/home/User/ApplicationName/esp-idf/make/project.mk:575: /home/User/ApplicationName/build/ApplicationName.elf] Error 1
kapilkedawat commented 3 years ago

I don't see any linking issue here, maybe retry after cleaning older project files? That may happen when os_xtensa.o is generated from the old config.

PaulFreund commented 3 years ago

Okay here is a small summary WITH the patch you posted here https://github.com/espressif/esp-idf/issues/7565#issuecomment-921641524, WITHOUT the patch from this comment https://github.com/espressif/esp-idf/issues/7384#issuecomment-913985517 and built with make:

ESP IDF version WPA Supplicant MBEDTLS enabled WPA2 Enterprise works Others
master true - Linker errors
master false true
tags/v4.3 true false
tags/v4.3 false true
PaulFreund commented 3 years ago

I always deleted my ./espressif and build directory before starting a new test to not get any old state. But maybe in that test the sdkconfig was slightly different.

Anyway, I think I can work with v4.3 + mbedtls disabled + your patch for now. Would be interresting why mbedtls does not work though

PaulFreund commented 3 years ago

@kapilkedawat Thank you so much! You saved my weekend :) 👍

PaulFreund commented 3 years ago

Because the application is quite universal I have to do a lot of testing before doing a release. Is it possible to backport this fix to release/v4.3?

kapilkedawat commented 3 years ago

Hi @PaulFreund , for the mbedTLS part, Is it possible for you to share this packet?

wpa: SSL: 146 bytes left to be sent out (of total 146 bytes) Excel line : 516, seems like this is causing AP to send a fail message.

Also if the server is reporting some error, please let me know.

Yes, we will backport this till v4.0 and all versions will have this fix.

PaulFreund commented 3 years ago

I will put it on my todo list, is it enough to enable verbose for wpa to get the data?

kapilkedawat commented 3 years ago

Yes, Please set both supplicant and MbedTLS to verbose level.

Alvin1Zhang commented 2 years ago

Thanks for reporting, fix on master branch is available at https://github.com/espressif/esp-idf/commit/71a5003ae74411e75056ec1d688cbd7e773df7a0.

AxelLin commented 2 years ago

I will put it on my todo list, is it enough to enable verbose for wpa to get the data?

@PaulFreund Any update with mbedTLS enabled?

AxelLin commented 2 years ago

@PaulFreund

Since it was working in 77eb201, this looks like a regression. Any chance to provide the log with mbedTLS enabled?

Alvin1Zhang commented 2 years ago

@PaulFreund Thanks for reporting, would you please help share if any further updates? Thanks.

Alvin1Zhang commented 2 years ago

Thanks for reporting, will close due to short of feedback, feel free to reopen with more updates.