search

esp-rs / esp-mbedtls

mbedtls for ESP32 bare-metal
Apache License 2.0
21 stars 10 forks source link

Examples broken at runtime. #23

Closed AnthonyGrondin closed 7 months ago

AnthonyGrondin commented 10 months ago

I've tried to run the following examples after a clean git clone and they all fail due to various issues:

EDIT: After further testing, the problem seems to occur on Xtensa targets. It had issues on esp32s3 but with esp32c3 everything is running without any failure.

Here's the output from running async_client:

Waiting to get IP address...
Got IP: 192.168.69.165/24
connecting...
Start tls connect

Exception occured 'InstrProhibited'
Context
PC=0x00000000       PS=0x00060a10
0x00000000 - XT_STK_PC
    at ??:??
0x00060a10 - PS_WOE
    at ??:??
A0=0x8037b2eb       A1=0x3fc9b900       A2=0x00000001       A3=0x00000000       A4=0x3fcbbfe0
0x8037b2eb - _rtc_fast_data_end
    at ??:??
0x3fc9b900 - _ZN8esp_wifi7preempt10TASK_STACK17hc849bc2d8dfdc35eE.llvm.8558853931319064652
    at ??:??
0x00000001 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x3fcbbfe0 - g_pm
    at ??:??
A5=0x3fcbbfe0       A6=0x3fcef940       A7=0x1d800160       A8=0x820a44aa       A9=0x00000000
0x3fcbbfe0 - g_pm
    at ??:??
0x3fcef940 - g_osi_funcs_p
    at ??:??
0x1d800160 - PS_WOE
    at ??:??
0x820a44aa - _rtc_fast_data_end
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
A10=0x00019000      A11=0x00000001      A12=0x3fc9b900      A13=0xeeeeeeef      A14=0x00000000
0x00019000 - RESERVE_ICACHE
    at ??:??
0x00000001 - XT_STK_PC
    at ??:??
0x3fc9b900 - _ZN8esp_wifi7preempt10TASK_STACK17hc849bc2d8dfdc35eE.llvm.8558853931319064652
    at ??:??
0xeeeeeeef - _rtc_fast_data_end
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
A15=0x00000006
0x00000006 - XT_STK_PS
    at ??:??
SAR=00000004
EXCCAUSE=0x00000014 EXCVADDR=0x00000000
0x00000014 - XT_STK_A3
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
LBEG=0x00000000     LEND=0x00000000     LCOUNT=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
THREADPTR=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
SCOMPARE1=0x00000001
0x00000001 - XT_STK_PC
    at ??:??
BR=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
ACCLO=0x00000000    ACCHI=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
M0=0x00000000       M1=0x00000000       M2=0x00000000       M3=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
F64R_LO=0x00000000  F64R_HI=0x00000000  F64S=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
FCR=0x00000000      FSR=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
F0=0x00000000       F1=0x00000000       F2=0x00000000       F3=0x00000000       F4=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
F5=0x00000000       F6=0x00000000       F7=0x00000000       F8=0x00000000       F9=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
F10=0x00000000      F11=0x00000000      F12=0x00000000      F13=0x00000000      F14=0x00000000
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
0x00000000 - XT_STK_PC
    at ??:??
F15=0x00000000
0x00000000 - XT_STK_PC
    at ??:??

0x4003d024
0x4003d024 - rom_rx_gain_force
    at ??:??
0x4037a49c
0x4037a49c - ppTask
    at ??:??
0x42021f84
0x42021f84 - core::sync::atomic::atomic_load
    at /home/anthony/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/sync/atomic.rs:3184
0x40000000
0x40000000 - ets_rom_layout_p
    at ??:??
bjoernQ commented 10 months ago

I cannot reproduce that exception on my side on ESP32-S3

I do see some TLS errors sometimes - but sometimes it just works

AnthonyGrondin commented 10 months ago

I'm on toolchain rustc 1.74.0-nightly (bd6bb31df 2023-11-15) (1.74.0.0)

Here's the steps to reproduce on my side. It's weird because it used to work at some point.

  1. Clone esp-mbedtls in a new directory

    git clone https://github.com/esp-rs/esp-mbedtls

  2. Run sync_server on esp32s3

    SSID=<SSID> PASSWORD=<SSID> cargo +esp run --example sync_server --target xtensa-esp32s3-none-elf -F esp32s3,esp-wifi/wifi-logs

  3. Panics on boot

    
INFO - esp-wifi configuration Config { rx_queue_size: 5, tx_queue_size: 3, static_rx_buf_num: 10, dynamic_rx_buf_num: 32, static_tx_buf_num: 0, dynamic_tx_buf_num: 32, ampdu_rx_enable: 0, ampdu_tx_enable: 0, amsdu_tx_enable: 0, rx_ba_win: 6, max_burst_size: 1, country_code: "CN", country_code_operating_class: 0, mtu: 1492, heap_size: 112640, tick_rate_hz: 50, listen_interval: 3, beacon_timeout: 6, ap_beacon_timeout: 300, failure_retry_cnt: 1, scan_method: 0 }

Exception occured 'LoadProhibited' Context PC=0x42069cf4 PS=0x00060930 0x42069cf4 - compiler_builtins::mem::impls::c_string_length at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.101/src/mem/impls.rs:286 0x00060930 - PS_WOE at ??:?? A0=0x8206abc1 A1=0x3fcd9e40 A2=0xdeadbeaf A3=0x000000ff A4=0x3fc8f886 0x8206abc1 - _rtc_fast_bss_start at ??:?? 0x3fcd9e40 - _stack_end at ??:?? 0xdeadbeaf - _rtc_fast_bss_start at ??:?? 0x000000ff - XT_STK_TMP at ??:?? 0x3fc8f886 - _ZN8esp_wifi14common_adapter12RADIO_CLOCKS17haa8afa80382b6914E at ??:?? A5=0x00000001 A6=0x00000001 A7=0x4201bbbc A8=0xdeadbeaf A9=0x00000001 0x00000001 - XT_STK_PC at ??:?? 0x00000001 - XT_STK_PC at ??:?? 0x4201bbbc - esp_wifi::preempt::preempt::task_create at ~/.cargo/git/checkouts/esp-wifi-836f3b2af57fa847/5166089/esp-wifi/src/preempt/preempt_xtensa.rs:69 0xdeadbeaf - _rtc_fast_bss_start at ??:?? 0x00000001 - XT_STK_PC at ??:?? A10=0x3fc8dbfc A11=0x00000001 A12=0x00000001 A13=0x00080002 A14=0x00380000 0x3fc8dbfc - _ZN14esp_hal_common21critical_section_impl9multicore14MULTICORE_LOCK17h0e1e74c9ce5dc8b1E at ??:?? 0x00000001 - XT_STK_PC at ??:?? 0x00000001 - XT_STK_PC at ??:?? 0x00080002 - PS_WOE at ??:?? 0x00380000 - PS_WOE at ??:?? A15=0x3fc8daa0 0x3fc8daa0 - _ZN8esp_wifi4wifi8G_CONFIG17h435367d823eeb9a0E at ??:?? SAR=00000010 EXCCAUSE=0x0000001c EXCVADDR=0xdeadbeaf 0x0000001c - XT_STK_A5 at ??:?? 0xdeadbeaf - _rtc_fast_bss_start at ??:?? LBEG=0x3c100b51 LEND=0x00000001 LCOUNT=0x00000004 0x3c100b51 - str.2 at ??:?? 0x00000001 - XT_STK_PC at ??:?? 0x00000004 - XT_STK_PS at ??:?? THREADPTR=0x00000000 0x00000000 - XT_STK_PC at ??:?? SCOMPARE1=0x00000001 0x00000001 - XT_STK_PC at ??:?? BR=0x00000000 0x00000000 - XT_STK_PC at ??:?? ACCLO=0x00000000 ACCHI=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? M0=0x00000000 M1=0x00000000 M2=0x00000000 M3=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? F64R_LO=0x86a00000 F64R_HI=0x00000001 F64S=0x00000000 0x86a00000 - _rtc_fast_bss_start at ??:?? 0x00000001 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? FCR=0x00000000 FSR=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? F0=0x00000000 F1=0x00000000 F2=0x00000000 F3=0x00000000 F4=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? F5=0x00000000 F6=0x00000000 F7=0x00000000 F8=0x00000000 F9=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? F10=0x00000000 F11=0x00000000 F12=0x00000000 F13=0x00000000 F14=0x00000000 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? 0x00000000 - XT_STK_PC at ??:?? F15=0x00000000 0x00000000 - XT_STK_PC at ??:??

0x4201a75b 0x4201a75b - core::ffi::c_str::CStr::from_ptr at ~/.rustup/toolchains/esp/lib/rustlib/src/rust/library/core/src/ffi/c_str.rs:273 0x4201a308 0x4201a308 - esp_wifi::compat::syslog::syslog at ??:?? 0x4201ae47 0x4201ae47 - coexist_printf at ~/.cargo/git/checkouts/esp-wifi-836f3b2af57fa847/5166089/esp-wifi/src/common_adapter/mod.rs:252 0x42092431 0x42092431 - wdev_funcs_init at ??:?? 0x4206db5e 0x4206db5e - esp_wifi_init_internal at ??:?? 0x42016fad 0x42016fad - esp_wifi::wifi::wifi_init at ~/.cargo/git/checkouts/esp-wifi-836f3b2af57fa847/5166089/esp-wifi/src/wifi/mod.rs:624 0x4201b9e5 0x4201b9e5 - esp_wifi::initialize at ~/.cargo/git/checkouts/esp-wifi-836f3b2af57fa847/5166089/esp-wifi/src/lib.rs:298 0x42008520 0x42008520 - sync_server::__xtensa_lx_rt_main at /tmp/esp-mbedtls/examples/sync_server.rs:55 0x42057017 0x42057017 - Reset at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/xtensa-lx-rt-0.16.0/src/lib.rs:70 0x403788a3 0x403788a3 - ESP32Reset at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/esp32s3-hal-0.13.0/src/lib.rs:

bjoernQ commented 10 months ago

Mhhh seems like in our 1.74 build the c-variadics are broken, again. e.g. for the first string it tries to get it e.g. reads the address 0xdeadbeaf ...

Works fine with 1.73.0.1 When not enabling wifi-logs it also works on 1.74

cc @MabezDev

MabezDev commented 10 months ago

I'm so sorry that's completely my bad. I forgot to add the commit to 1.74, because I was looking at the patch set from 1.73.0.0, instead of 1.73.0.1 🤦. I'll fix that and get a release out soon.

AnthonyGrondin commented 10 months ago

Alright, I'm using rustc 1.73.0-nightly (2fba83302 2023-10-18) (1.73.0.1) to fix the issue mentionned above.

Examples run without any issue on the esp32c3. I'm facing issues running them on the esp32s3.

Now, I'm only facing runtime issues during a connection. When I run the sync_server example like follow:

SSID=<SSID> PASSWORD=<PASS> cargo +esp run --release --example sync_server --target xtensa-esp32s3-none-elf -F esp32s3

I get the following error during a connection:

INFO - esp-wifi configuration Config { rx_queue_size: 10, tx_queue_size: 10, static_rx_buf_num: 10, dynamic_rx_buf_num: 32, static_tx_buf_num: 0, dynamic_tx_buf_num: 32, ampdu_rx_enable: 0, ampdu_tx_enable: 0, amsdu_tx_enable: 0, rx_ba_win: 6, max_burst_size: 1, country_code: "CN", country_code_operating_class: 0, mtu: 1492, heap_size: 112640, tick_rate_hz: 100, listen_interval: 3, beacon_timeout: 6, ap_beacon_timeout: 300, failure_retry_cnt: 1, scan_method: 0 }
Call wifi_connect
Wait to get connected
Wait to get an ip address
Got ip Ok(IpInfo { ip: 192.168.69.165, subnet: Subnet { gateway: 192.168.69.1, mask: Mask(24) }, dns: Some(192.168.69.16), secondary_dns: None })
We are connected!
Point your browser to https://192.168.69.165/
New connection

!! A panic occured in 'examples/sync_server.rs', at line 198, column 21

PanicInfo {
    payload: Any { .. },
    message: Some(
        MbedTlsError(-17280),
    ),
    location: Location {
        file: "examples/sync_server.rs",
        line: 198,
        col: 21,
    },
    can_unwind: true,
}

The error code is for MBEDTLS_ERR_RSA_VERIFY_FAILED. The same example works on the C3, so I don't see why it fails for the S3

bjoernQ commented 10 months ago

I can reproduce it. I also tried different versions of the toolchain and very few different (recent) commits of esp-mbedtls Also, different opt-levels don't seem to make a difference (other than timing out with level 1)

Oh wait ... seems like a484bba6375d23c4e47cbf342ca18afda44482b1 works for me (but needs a local esp-wifi rev 68dc11bbb2c0efa29c4acbbf134d6f142441065e and changes in Software1 and yield_task)

Aaaaand I can also get the current main to work when copying the pre-compiled libs from a484bba6375d23c4e47cbf342ca18afda44482b1

If it's reproducible on your side then there is something wrong with how mbedtls is built for Xtensa while it's fine for RISC-V

AnthonyGrondin commented 10 months ago

I've tested with the pre-compiled libraries from before we refactored the build scripts to use xtasks and it works.

The issue effectively seems to be from the way we build mbedtls for Xtensa, introduced in #15

AnthonyGrondin commented 7 months ago

I'm going to close this because it has been fixed in #24 after recompiling using clang