search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.88k stars 7.32k forks source link

mbedTLS memory allocation stragegy SPIRAM/default allocation mode breaks mbedTLS with gcc 8 (IDFGH-1336) #3624

Closed PerMalmberg closed 3 years ago

PerMalmberg commented 5 years ago

Environment

Problem Description

This bug relates to this question of the forum.

To make a long story short - forcing mbedTLS to use external memory breaks it in such a way that all TLS handshakes results in this error:

mbedtls_ssl_handshake returned -29056

meaning "SSL - Verification of the message MAC failed"

Expected Behavior

TLS to be functional even when using SPIRAM

Actual Behavior

I (15503) tcpip_adapter: sta ip: 192.168.10.94, mask: 255.255.255.0, gw: 192.168.10.254 I (15503) sslServer: Connected to AP E (26173) sslServer: failed ! mbedtls_ssl_handshake returned -29056

Steps to repropduce

I've attached a zip with a modified https_server example in which I've replaced the actual server with one that uses mbedTLS (original source by @projectgus here, but modified to use the certs provided in the https_server project).

Simply change the hard-coded "CHANGE ME" ssid and password in main2.c, compile, run and browse to IP:9080 and you'll see your browser emit something like this, as well as the above log message.

An error occurred during a connection to 192.168.10.94:9080. SSL peer reports incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_ALERT

Changing mbedTLS memory allocation strategy to internal makes the handshake function again and the server will output the browsers request on screen.

I've intentionally forced most memory allocations to use SPIRAM by setting CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL to 256 to ensure the problem is shown in this example.

Unfortunately there is not enough internal memory for my project so I can't do without SPIRAM so any attempt to have mbedTLS only use internal memory results in memory allocation errors, with the side effects that brings.

mbedtls_broken_with_gcc8.zip

negativekelvin commented 5 years ago

In a browser it fails but it works in curl. Actually it works in a browser as long as you force https in the URL. The problem with chrome is that it will timeout on the first try and close the connection but it will succeed on the second try. However your example code returns on handshake failure instead of continuing the accept loop.

PerMalmberg commented 5 years ago

@negativekelvin Then you're getting other results than me. Just verified that if I browse to https://IP:9080, then I get the MAC-failure. Did you use the sdkconfig in the zip?

ff

However your example code returns on handshake failure instead of continuing the accept loop.

It was the only example that uses mbedTLS I found that wasn't written by me I could find and in this case it doesn't matter - the MAC failure happens.

negativekelvin commented 5 years ago

Yes I only changed the Serial flasher settings.

Firefox:

I (1098) sslServer: Connected to AP
Result: [size: 339]
GET / HTTP/1.1
Host: 192.168.1.50:9080
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Fir
efox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Chrome:

E (1775488) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592

Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.1.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/73.0.3683.103 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/a
png,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
PerMalmberg commented 5 years ago

Could the difference in result be due to different hardware? What device are you running on, @negativekelvin ?

negativekelvin commented 5 years ago

No same hardware but I am using make and gcc 5.

PerMalmberg commented 5 years ago

Ah, that might be the difference then. Could you please try with gcc 8 and same IDF just to see if it is reproducible by others than me?

negativekelvin commented 5 years ago

Yes switching to gcc 8 I can reproduce it

It does get pretty far into the handshake and then mbedtls_cipher_auth_decrypt() returned -25344 (-0x6300)

PerMalmberg commented 5 years ago

@negativekelvin Thanks for confirming.

So it seems this is related to the gcc 8 tool chain and thus of interest for @igrr perhaps?

PerMalmberg commented 5 years ago

For the record: I tried the very latest IDF and xtensa-gcc 8 with the same result. OP updated with new versions.

Also, I tried to go back to gcc 5, but gcc 8 rc1 is the only one offered on the download page now so I'm totally blocked now.

igrr commented 5 years ago

For the GCC 5 link, you can still find it here: https://docs.espressif.com/projects/esp-idf/en/stable/get-started/linux-setup.html#toolchain-setup

PerMalmberg commented 5 years ago

Oh right. I always go /latest in the docs, not /stable Thank you @igrr

PerMalmberg commented 5 years ago

I forgot - with access to gcc 8 I've started using C++ 17, so going back to gcc 5 isn't an appealing option. Using bleeding edge tech is fun, but sometimes it hurts :hankey: Then again, that's what early access is for.

xiruilin commented 5 years ago

Use "libc-psram-workaround.a" instead of "libc.a" to solve the problem, https://github.com/espressif/arduino-esp32/issues/2080 But now gcc 8.2.0 don't support "libc-psram-workaround.a", error message:

I (754) cpu_start: Starting app cpu, entry point is 0x400814a4 I (0) cpu_start: App cpu up. I (1234) spiram: SPI SRAM memory test OK I (1236) heap_init: Initializing. RAM available for dynamic allocation: I (1236) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM I (1241) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM I (1247) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM I (1253) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM I (1259) heap_init: At 3FFC4740 len 0001B8C0 (110 KiB): DRAM I (1266) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM I (1272) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM I (1279) heap_init: At 40099E50 len 000061B0 (24 KiB): IRAM I (1285) cpu_start: Pro cpu start user code I (1290) spiram: Adding pool of 4070K of external SPI memory to heap allocator I (1311) pm_esp32: Frequency switching config: CPU_MAX: 240, APB_MAX: 240, APB_MIN: 40, Light sleep: DISABLED I (1311) cpu_start: Starting scheduler on PRO CPU. I (0) cpu_start: Starting scheduler on APP CPU. I (1325) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations GGuru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled. Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad uGuru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled. Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad Guru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled. Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad

negativekelvin commented 5 years ago

@xiruilin suggestion works, overriding the newlib component with the gcc8 versions.

Actually I take it back it doesn't work 100% of the time. It might require the new psram fix.

PerMalmberg commented 5 years ago

It might require the new psram fix

You mean the outstanding issue in gcc 8?

negativekelvin commented 5 years ago

I mean this https://github.com/espressif/esp-idf/issues/2892

PerMalmberg commented 5 years ago

I mean this #2892

Yeah, that one. I wonder though - this MAC issue seems to only occur in gcc 8. Why is that you think? gcc8 just happens to generate code that triggers the PSRAM every time it runs the crypto code?

negativekelvin commented 5 years ago

Well I did not test enough to say it doesn't ever occur in gcc 5. I don't know if there are ABI changes that cause exactly the right trigger, but maybe.

PerMalmberg commented 5 years ago

Tried with the latest IDF (original post updated), and the issue remains.

PerMalmberg commented 5 years ago

It's been a month since this ticket was opened, would it be possible to get some feedback from Espressif, @igrr?

enricop commented 5 years ago

Hi all, we have the same issue in our project, where we have memory constraints. Before the recent upgrade to mbedtls 2.16.2 the allocation into SPI RAM was working. But now error occurs in tls handshake. (We are using W-Rover-B and GCC8)

igrr commented 5 years ago

@PerMalmberg with the latest master branch (842432fb654ebabd04b2fbaefad68ea67ee4fef6) I can not reproduce the issue using the project attached to the OP. I see a handshake error -0x7780 with Chrome (as @negativekelvin has reported) and both GCC 5.2 and 8.2 based toolchains. There is no error with curl and Safari in both GCC versions.

However, while looking into this issue, I have found a potential problem related to the way psram-workaround version of libc.a was built for the GCC 8 toolchain. Could you please try replacing /path-to/xtensa-esp32-elf/xtensa-esp32-elf/lib/esp32-psram/libc.a with the attached copy, and see if the issue persists?

libc.a.zip (the archive was stripped of debug symbols to make it fit into an attachment)

PerMalmberg commented 5 years ago 
[827/828] Linking CXX executable https_server.elf
FAILED: https_server.elf 
: && /home/permal/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++  -mlongcalls -Wno-frame-address  -nostdlib CMakeFiles/https_server.elf.dir/project_elf_src.c.obj  -o https_server.elf  esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/driver/libdriver.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/efuse/libefuse.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/app_update/libapp_update.a esp-idf/spi_flash/libspi_flash.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/lwip/liblwip.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/esp_event/libesp_event.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/esp32/libesp32.a esp-idf/xtensa/libxtensa.a esp-idf/esp_common/libesp_common.a esp-idf/esp_rom/libesp_rom.a esp-idf/soc/libsoc.a esp-idf/log/liblog.a esp-idf/heap/libheap.a esp-idf/freertos/libfreertos.a esp-idf/vfs/libvfs.a esp-idf/newlib/libnewlib.a esp-idf/cxx/libcxx.a esp-idf/app_trace/libapp_trace.a esp-idf/asio/libasio.a esp-idf/coap/libcoap.a esp-idf/console/libconsole.a esp-idf/nghttp/libnghttp.a esp-idf/esp-tls/libesp-tls.a esp-idf/esp_adc_cal/libesp_adc_cal.a esp-idf/esp_gdbstub/libesp_gdbstub.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/esp_https_ota/libesp_https_ota.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/protocomm/libprotocomm.a esp-idf/mdns/libmdns.a esp-idf/esp_local_ctrl/libesp_local_ctrl.a esp-idf/esp_websocket_client/libesp_websocket_client.a esp-idf/expat/libexpat.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/fatfs/libfatfs.a esp-idf/freemodbus/libfreemodbus.a esp-idf/jsmn/libjsmn.a esp-idf/json/libjson.a esp-idf/libsodium/liblibsodium.a esp-idf/mqtt/libmqtt.a esp-idf/openssl/libopenssl.a esp-idf/spiffs/libspiffs.a esp-idf/ulp/libulp.a esp-idf/unity/libunity.a esp-idf/wifi_provisioning/libwifi_provisioning.a esp-idf/main/libmain.a esp-idf/protocol_examples_common/libprotocol_examples_common.a -Wl,--cref -Wl,--Map=/home/permal/tmp/build/https_server.map esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/driver/libdriver.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/efuse/libefuse.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/app_update/libapp_update.a esp-idf/spi_flash/libspi_flash.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/lwip/liblwip.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/esp_event/libesp_event.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/esp32/libesp32.a esp-idf/xtensa/libxtensa.a esp-idf/esp_common/libesp_common.a esp-idf/esp_rom/libesp_rom.a esp-idf/soc/libsoc.a esp-idf/log/liblog.a esp-idf/heap/libheap.a esp-idf/freertos/libfreertos.a esp-idf/vfs/libvfs.a esp-idf/newlib/libnewlib.a esp-idf/cxx/libcxx.a esp-idf/app_trace/libapp_trace.a esp-idf/asio/libasio.a esp-idf/coap/libcoap.a esp-idf/console/libconsole.a esp-idf/nghttp/libnghttp.a esp-idf/esp-tls/libesp-tls.a esp-idf/esp_adc_cal/libesp_adc_cal.a esp-idf/esp_gdbstub/libesp_gdbstub.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/esp_https_ota/libesp_https_ota.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/protocomm/libprotocomm.a esp-idf/mdns/libmdns.a esp-idf/esp_local_ctrl/libesp_local_ctrl.a esp-idf/esp_websocket_client/libesp_websocket_client.a esp-idf/expat/libexpat.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/fatfs/libfatfs.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/freemodbus/libfreemodbus.a esp-idf/jsmn/libjsmn.a esp-idf/json/libjson.a esp-idf/libsodium/liblibsodium.a esp-idf/mqtt/libmqtt.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp-tls/libesp-tls.a esp-idf/openssl/libopenssl.a esp-idf/spiffs/libspiffs.a esp-idf/ulp/libulp.a esp-idf/unity/libunity.a esp-idf/wifi_provisioning/libwifi_provisioning.a esp-idf/protocomm/libprotocomm.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/nghttp/libnghttp.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/mdns/libmdns.a esp-idf/console/libconsole.a esp-idf/json/libjson.a esp-idf/cxx/libcxx.a esp-idf/newlib/libnewlib.a esp-idf/freertos/libfreertos.a esp-idf/heap/libheap.a esp-idf/log/liblog.a esp-idf/soc/libsoc.a esp-idf/esp_rom/libesp_rom.a esp-idf/esp_common/libesp_common.a esp-idf/xtensa/libxtensa.a esp-idf/esp32/libesp32.a esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/lwip/liblwip.a esp-idf/mbedtls/mbedtls/library/libmbedtls.a esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a esp-idf/mbedtls/mbedtls/library/libmbedx509.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/spi_flash/libspi_flash.a esp-idf/efuse/libefuse.a esp-idf/app_update/libapp_update.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcore.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libespnow.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libmesh.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libphy.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libpp.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/librtc.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a esp-idf/esp_event/libesp_event.a esp-idf/driver/libdriver.a esp-idf/vfs/libvfs.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/app_trace/libapp_trace.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/cxx/libcxx.a esp-idf/newlib/libnewlib.a esp-idf/freertos/libfreertos.a esp-idf/heap/libheap.a esp-idf/log/liblog.a esp-idf/soc/libsoc.a esp-idf/esp_rom/libesp_rom.a esp-idf/esp_common/libesp_common.a esp-idf/xtensa/libxtensa.a esp-idf/esp32/libesp32.a esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/lwip/liblwip.a esp-idf/mbedtls/mbedtls/library/libmbedtls.a esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a esp-idf/mbedtls/mbedtls/library/libmbedx509.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/spi_flash/libspi_flash.a esp-idf/efuse/libefuse.a esp-idf/app_update/libapp_update.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcore.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libespnow.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libmesh.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libphy.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libpp.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/librtc.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a esp-idf/esp_event/libesp_event.a esp-idf/driver/libdriver.a esp-idf/vfs/libvfs.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/app_trace/libapp_trace.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a -lstdc++ -u __cxa_guard_dummy -u __cxx_fatal_exception esp-idf/newlib/libnewlib.a -u newlib_include_locks_impl -u newlib_include_heap_impl -u newlib_include_syscalls_impl -u newlib_include_pthread_impl -Wl,--undefined=uxTopUsedPriority -L /home/permal/esp/esp-idf/components/esp_rom/esp32/ld -T esp32.rom.ld -T esp32.rom.libgcc.ld -T esp32.rom.syscalls.ld -T esp32.rom.newlib-data.ld -Wl,--gc-sections /home/permal/esp/esp-idf/components/xtensa/esp32/libhal.a -L /home/permal/tmp/build/esp-idf/esp32 -T esp32_out.ld -u app_main -L /home/permal/esp/esp-idf/components/esp32/ld -T esp32.extram.bss.ld -L /home/permal/tmp/build/esp-idf/esp32/ld -T esp32.project.ld -T esp32.peripherals.ld -u call_user_start_cpu0 -u ld_include_panic_highint_hdl -mfix-esp32-psram-cache-issue -u esp_app_desc -u vfs_include_syscalls_impl -L /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32 -lgcov -lc -lm -lgcc && :
/home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/bin/ld: /home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/lib/esp32-psram/libc.a: error adding symbols: archive has no index; run ranlib to add one
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed

Running ar -t libc.a yields a list of several object files:

ar -t libc.a                                                                                                                                     
_atexit.o
clibrary_init.o
creat.o
isatty.o
lib_a-_Exit.o
lib_a-__adjust.o
lib_a-__atexit.o
lib_a-__call_atexit.o
lib_a-__dprintf.o
...continued...
igrr commented 5 years ago

Sorry, stripping the static library was obviously a bad idea. The full version is at https://dl.espressif.com/dl/misc/IDFGH-1336-libc.a.zip, please give a try.

PerMalmberg commented 5 years ago

Full test results:

I wonder why you can't reproduce it with the latest master and the original libc.a ?

negativekelvin commented 5 years ago

for me it is still not working consistently

D (4970) sslServer: >> waiting for accept
E (9360) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592

D (9370) sslServer: >> waiting for accept
E (11720) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592

D (11730) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

D (15780) sslServer: >> waiting for accept
E (102280) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592

D (102280) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

D (106310) sslServer: >> waiting for accept
E (113650) sslServer:  failed
  ! mbedtls_ssl_handshake returned -29312

D (113650) sslServer: >> waiting for accept
E (115690) sslServer:  failed
  ! mbedtls_ssl_handshake returned -78

D (115700) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

D (119520) sslServer: >> waiting for accept
E (127220) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592

D (127230) sslServer: >> waiting for accept
E (130710) sslServer:  failed
  ! mbedtls_ssl_handshake returned -29056

D (130720) sslServer: >> waiting for accept
I (134830) sslServer: connection closed
Result: [size: 0]
xiruilin commented 5 years ago

@enricop If you got a handshake error after upgrade to mbedtls 2.16.2, do below in "esp-idf/components/mbedtls/port/net_sockets.c", line 209:

origin: int error = mbedtls_net_errno(ctx->fd);

change to: int error = errno;

negativekelvin commented 5 years ago

@PerMalmberg is it still broken or am I doing something wrong?

PerMalmberg commented 5 years ago

As per my last post, it now works for me, this is the entire output:

rst:0x1 (POWERON_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371 
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x3e (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:6556
load:0x40078000,len:12116
load:0x40080400,len:7172
entry 0x40080770
I (29) boot: ESP-IDF v4.0-dev-1136-g28f1cdf5e 2nd stage bootloader
I (29) boot: compile time 12:25:30
I (31) boot: Enabling RNG early entropy source...
I (35) boot: SPI Speed      : 40MHz
I (39) boot: SPI Mode       : DIO
I (43) boot: SPI Flash Size : 4MB
I (47) boot: Partition Table:
I (51) boot: ## Label            Usage          Type ST Offset   Length
I (58) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (66) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (73) boot:  2 factory          factory app      00 00 00010000 00300000
I (81) boot:  3 app_storage      Unknown data     01 81 00310000 00084000
I (88) boot: End of partition table
I (92) esp_image: segment 0: paddr=0x00010020 vaddr=0x3f400020 size=0x21154 (135508) map
I (150) esp_image: segment 1: paddr=0x0003117c vaddr=0x3ffb0000 size=0x031d8 ( 12760) load
I (155) esp_image: segment 2: paddr=0x0003435c vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /home/permal/esp/esp-idf/components/freertos/xtensa_vectors.S:1778

I (157) esp_image: segment 3: paddr=0x00034764 vaddr=0x40080400 size=0x0b8ac ( 47276) load
I (185) esp_image: segment 4: paddr=0x00040018 vaddr=0x400d0018 size=0x937d0 (604112) map
0x400d0018: _stext at ??:?

I (402) esp_image: segment 5: paddr=0x000d37f0 vaddr=0x4008bcac size=0x093a8 ( 37800) load
0x4008bcac: _isatty_r at /builds/c7bbafe1/0/idf/newlib-cygwin/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/sys/xtensa/isatty.c:15

I (431) boot: Loaded app from partition at offset 0x10000
I (431) boot: Disabling RNG early entropy source...
I (431) psram: This chip is ESP32-D0WD
I (437) spiram: Found 32MBit SPI RAM device
I (440) spiram: SPI RAM mode: flash 40m sram 40m
I (446) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (453) cpu_start: Pro cpu up.
I (457) cpu_start: Application information:
I (462) cpu_start: Project name:     https_server
I (467) cpu_start: App version:      1
I (471) cpu_start: Compile time:     Jul 27 2019 10:15:26
I (477) cpu_start: ELF file SHA256:  7630a19a860cfea3...
I (483) cpu_start: ESP-IDF:          v4.0-dev-1338-g842432fb6
I (490) cpu_start: Starting app cpu, entry point is 0x400813b0
0x400813b0: call_start_cpu1 at /home/permal/esp/esp-idf/components/esp32/cpu_start.c:283

I (0) cpu_start: App cpu up.
I (1391) spiram: SPI SRAM memory test OK
I (1393) heap_init: Initializing. RAM available for dynamic allocation:
I (1394) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (1398) heap_init: At 3FFB4C48 len 0002B3B8 (172 KiB): DRAM
I (1405) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (1411) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (1418) heap_init: At 40095054 len 0000AFAC (43 KiB): IRAM
I (1424) cpu_start: Pro cpu start user code
I (1429) spiram: Adding pool of 4078K of external SPI memory to heap allocator
I (1451) spi_flash: detected chip: generic
I (1451) spi_flash: flash io: dio
I (1451) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
I (1459) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (1509) wifi: wifi driver task: 3ffc40ac, prio:23, stack:3584, core=0
I (1509) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (1509) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (1539) wifi: wifi firmware version: fc2a27d
I (1539) wifi: config NVS flash: enabled
I (1539) wifi: config nano formating: disabled
I (1539) wifi: Init dynamic tx buffer num: 32
I (1539) wifi: Init data frame dynamic rx buffer num: 32
I (1549) wifi: Init management frame dynamic rx buffer num: 32
I (1549) wifi: Init management short buffer num: 32
I (1559) wifi: Init static tx buffer num: 16
I (1559) wifi: Init static rx buffer size: 1600
I (1559) wifi: Init static rx buffer num: 10
I (1569) wifi: Init dynamic rx buffer num: 32
I (1569) sslServer: Setting WiFi configuration SSID Our place...
I (1659) phy: phy_version: 4102, 2fa7a43, Jul 15 2019, 13:06:06, 0, 0
I (1659) wifi: mode : sta (84:0d:8e:18:8b:a0)
I (1789) wifi: new:<1,1>, old:<1,0>, ap:<255,255>, sta:<1,1>, prof:1
I (2539) wifi: state: init -> auth (b0)
I (3549) wifi: state: auth -> init (200)
I (3549) wifi: new:<1,0>, old:<1,1>, ap:<255,255>, sta:<1,1>, prof:1
I (5719) wifi: new:<1,1>, old:<1,0>, ap:<255,255>, sta:<1,1>, prof:1
I (5719) wifi: state: init -> auth (b0)
I (5729) wifi: state: auth -> assoc (0)
I (5739) wifi: state: assoc -> run (10)
I (5759) wifi: connected with Our place, channel 1, 40U, bssid = f0:9f:c2:aa:b0:3f
I (5759) wifi: pm start, type: 1

I (7999) tcpip_adapter: sta ip: 192.168.10.94, mask: 255.255.255.0, gw: 192.168.10.254
I (7999) sslServer: Connected to AP
Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Result: [size: 338]
GET / HTTP/1.1
Host: 192.168.10.94:9080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

I (59119) sslServer: connection closed
Result: [size: 0]
negativekelvin commented 5 years ago

Ok it is working for me but only with cmake, under make I assume it is not linking against the proper toolchain libraries. It also doesn't seem to work under cmake with custom toolchain prefix (path) that is not in PATH.

igrr commented 5 years ago

Thanks @negativekelvin, it seems that LDFLAGS+=-mfix-esp32-psram-cache-issue is missing from components/esp32/Makefile.projbuild, therefore incorrect multilib configuration is used when linking. Will fix.

with custom toolchain prefix (path) that is not in PATH

Sorry, can you explain this one a bit more? Do you mean that you are setting CONFIG_SDK_TOOLPREFIX to include the absolute path to the toolchain?.. If this is the case, is there another copy of the toolchain which is found in the PATH?

negativekelvin commented 5 years ago

Yes gcc5 was at opt/xtensa-esp32-elf/bin which is in PATH and toolchain prefix for gcc8 was set to /opt/xtensa-esp32-elf.gcc8/bin/xtensa-esp32-elf- I'm not sure how the multilib directory is located by the toolchain

PerMalmberg commented 5 years ago

Can we expect an updated tool chain with this fix anytime soon?

PerMalmberg commented 5 years ago

@igrr Where are we on this? Any more changes coming, or will it be in the next release of the tool chain?

PerMalmberg commented 5 years ago

Another one and a half month later, status update, please @igrr ?

igrr commented 5 years ago

@PerMalmberg sorry, the fix should indeed be part of the new toolchain release. Unfortunately it got held up by a different issue, which we are working on. Hopefully it will get solved next week.

enricop commented 5 years ago

yes, all this seems related to a bigger issue concerning external ram allocation, which corrupts in some way. We wait for this fix for our project too! thanks for your support

jeromeDms commented 5 years ago

We also wait for this fix :)

AndroidBaaaby commented 5 years ago

Hi @igrr , I was just wondering when the new toolchain will be released.

igrr commented 5 years ago

Unfortunately we ran into an issue with enabling RTTI support in this new toolchain build. We are working on some alternative solution. I will update this issue when we have some progress!

igrr commented 5 years ago

The toolchain has been updated in https://github.com/espressif/esp-idf/commit/c2db6a14c4404c6f50e096df459a096b47253f4a#diff-0633a6582c888c6fc5bc6a3c00dba496, this version includes the correctly built libc.a. Please update the toolchain and give it a try. This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

mcilloni commented 5 years ago

The toolchain has been updated in c2db6a1#diff-0633a6582c888c6fc5bc6a3c00dba496, this version includes the correctly built libc.a. Please update the toolchain and give it a try. This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

Will release/4.0 also support this new toolchain? We're currently using the betas from 4.0 on our projects and it would be nice to avoid having to swap libc.a with the patched one.

PerMalmberg commented 5 years ago

This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

Which, apart from possibly solving the issue reported in this thread, still leaves us with only unicore being a stable mode of operation. Can we expect another updated toolchain before the final release of v4.0 ?

igrr commented 5 years ago

release/v4.0 has also been updated to this toolchain release, it will get synced to GitHub once we fix some of the failing tests on our side.

Regarding the new PSRAM workaround version, I think it is unlikely that it will be finished in time for the 4.0 release. We will definitely include it into the subsequent patch release (4.0.x), once this toolchain version is ready. The linked issue will be updated when this happens.

sebirdman commented 4 years ago

@igrr using the latest release/v4.0 ( a3f3c7b ) with the 2020r1 psram fix toolchain release still results in this bug. Are there other steps i should be taking or should i to wait for release/4.0 to be updated?

edit:

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs) seem to make the problem go away. I'm using the idf_as_lib strategy with cmake for building. Is there some way i should be properly linking these? I have:

set(CMAKE_C_FLAGS "-mlongcalls -mfix-esp32-psram-cache-issue -Wno-frame-address" CACHE STRING "C Compiler Base Flags")
set(CMAKE_CXX_FLAGS "-mlongcalls -mfix-esp32-psram-cache-issue -Wno-frame-address" CACHE STRING "C++ Compiler Base Flags")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "shared ldflags")
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "module linker flags")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "exe linker flags")

in my toolchain.cmake file

igrr commented 4 years ago

Could you please do a verbose build, copy the final linker command line, and paste it here?

sebirdman commented 4 years ago

@igrr the final command is the one that generates the .elf file correct?

i notice that the espressif components have the -mfix-esp32-psram-cache flag after the .a blobs, but mine do not. Maybe that's part of the issue.

xtensa-esp32-elf-g++  -mlongcalls -Wno-frame-address  -mfix-esp32-psram-cache-issue CMakeFiles/app.elf.dir/main/app/main.c.obj  -o app.elf  source/libs/home_controller/libhome_controller.a  source/libs/global_state/libglobal_state.a  esp-idf/esp32/libesp32.a  -mfix-esp32-psram-cache-issue  -fno-rtti  -fno-lto  source/libs/monitors/libmonitors.a  source/libs/shopvac/libshopvac.a  source/libs/logdna_vacuum/liblogdna_vacuum.a  source/libs/file_ringbuffer/libfile_ringbuffer.a  source/libs/coredump_vacuum/libcoredump_vacuum.a  platform/esp32/port/coredump_platform/libcoredump_platform_esp32.a  source/libs/stats_vacuum/libstats_vacuum.a  source/libs/debug_shell/libdebug_shell.a  source/libs/plugin_manager/libplugin_manager.a  source/libs/diagnostics/libdiagnostics.a  source/libs/bluetooth_state/libbluetooth_state.a  source/libs/global_state/libglobal_state.a  source/hw/i2c/libi2c.a  source/libs/company_log/libcompany_log.a  source/libs/e2e/libe2e.a  source/libs/jobs/libjobs.a  source/libs/corewifi/libcorewifi.a  source/libs/internet_manager/libinternet_manager.a  source/libs/company_ui/libcompany_ui.a  source/libs/time_machine/libtime_machine.a  source/libs/aquaman/libaquaman.a  source/hw/hardware/libhardware.a  source/hw/kvs/libkvs.a  source/hw/storage/libstorage.a  source/libs/company_kvs/libcompany_kvs.a  source/libs/company_encryption/libcompany_encryption.a  source/hw/hwinfo/libhwinfo.a  source/libs/event_manager/libevent_manager.a  source/libs/ocelot/libocelot.a  source/hw/platform/libplatform.a  source/libs/sentry_client/libsentry_client.a  source/libs/enum_to_string/libenum_to_string.a  source/libs/profile/libprofile.a  source/libs/mqtt_setup/libmqtt_setup.a  source/libs/company_svcs/libcompany_svcs.a  source/libs/bt_handler/libbt_handler.a  source/libs/corebt/libcorebt.a  platform/esp32/port/bt_platform/libbt_platform_esp32.a  dependencies/generic/telnet_client/liblibtelnet.a  platform/esp32/port/wifi_platform/libwifi_platform.a  source/libs/mqtt_client/libmqtt_client.a  source/libs/local_api/liblocal_api.a  source/libs/company_packet_processor/libcompany_packet_processor.a  source/libs/lytebyte/liblytebyte.a  source/hw/gpio/libgpio.a  platform/esp32/port/kvs_platform/libkvs_platform.a  platform/esp32/port/storage_platform/libstorage_platform.a  source/libs/uuid4/libuuid4.a  platform/esp32/port/i2c_platform/libi2c_platform.a  platform/esp32/port/platform/libplatform_platform.a  source/libs/https_manager/libhttps_manager.a  source/libs/company_json/libcompany_json.a  source/libs/firmware_update/libfirmware_update.a  source/libs/wifi_provisioner/libwifi_provisioner.a  source/libs/bt_mesh_stack/libbt_mesh_stack.a  source/libs/jwt/libjwt.a  source/libs/mesh_handlers/libmesh_handlers.a  source/libs/device_info_service/libdevice_info_service.a  source/libs/linking_handler/liblinking_handler.a  source/libs/s3_utils/libs3_utils.a  platform/esp32/port/ping_platform/libping_platform.a  dependencies/generic/paho/liblibpaho.a  source/libs/sddp/libsddp.a  platform/esp32/port/rgbw_platform/librgbw_platform.a  platform/esp32/port/timer_platform/libtimer_platform.a  source/libs/b64url/libb64url.a  platform/esp32/port/https_requests_platform/libhttps_requests_platform_esp32.a  platform/esp32/port/ota_platform/libota_platform.a  source/libs/global_state/libglobal_state.a  source/hw/i2c/libi2c.a  source/libs/company_log/libcompany_log.a  source/libs/e2e/libe2e.a  source/libs/jobs/libjobs.a  source/libs/corewifi/libcorewifi.a  source/libs/internet_manager/libinternet_manager.a  source/libs/company_ui/libcompany_ui.a  source/libs/time_machine/libtime_machine.a  source/libs/aquaman/libaquaman.a  source/hw/hardware/libhardware.a  source/hw/kvs/libkvs.a  source/hw/storage/libstorage.a  source/libs/company_kvs/libcompany_kvs.a  source/libs/company_encryption/libcompany_encryption.a  source/hw/hwinfo/libhwinfo.a  source/libs/event_manager/libevent_manager.a  source/libs/ocelot/libocelot.a  source/hw/platform/libplatform.a  source/libs/sentry_client/libsentry_client.a  source/libs/enum_to_string/libenum_to_string.a  source/libs/profile/libprofile.a  source/libs/mqtt_setup/libmqtt_setup.a  source/libs/company_svcs/libcompany_svcs.a  source/libs/bt_handler/libbt_handler.a  source/libs/corebt/libcorebt.a  platform/esp32/port/bt_platform/libbt_platform_esp32.a  dependencies/generic/telnet_client/liblibtelnet.a  platform/esp32/port/wifi_platform/libwifi_platform.a  source/libs/mqtt_client/libmqtt_client.a  source/libs/local_api/liblocal_api.a  source/libs/company_packet_processor/libcompany_packet_processor.a  source/libs/lytebyte/liblytebyte.a  source/hw/gpio/libgpio.a  platform/esp32/port/kvs_platform/libkvs_platform.a  platform/esp32/port/storage_platform/libstorage_platform.a  source/libs/uuid4/libuuid4.a  platform/esp32/port/i2c_platform/libi2c_platform.a  platform/esp32/port/platform/libplatform_platform.a  source/libs/https_manager/libhttps_manager.a  source/libs/company_json/libcompany_json.a  source/libs/firmware_update/libfirmware_update.a  source/libs/wifi_provisioner/libwifi_provisioner.a  source/libs/bt_mesh_stack/libbt_mesh_stack.a  source/libs/jwt/libjwt.a  source/libs/mesh_handlers/libmesh_handlers.a  source/libs/device_info_service/libdevice_info_service.a  source/libs/linking_handler/liblinking_handler.a  source/libs/s3_utils/libs3_utils.a  platform/esp32/port/ping_platform/libping_platform.a  dependencies/generic/paho/liblibpaho.a  source/libs/sddp/libsddp.a  platform/esp32/port/rgbw_platform/librgbw_platform.a  platform/esp32/port/timer_platform/libtimer_platform.a  source/libs/b64url/libb64url.a  platform/esp32/port/https_requests_platform/libhttps_requests_platform_esp32.a  platform/esp32/port/ota_platform/libota_platform.a  platform/esp32/port/ntp_platform/libntp_platform.a  platform/esp32/port/ntp_platform/libntp_platform.a  dependencies/generic/addons/libfreertos_extras.a  source/libs/json_stream/libjson_stream.a  esp-idf/bt/libbt.a  -L/esp-idf/components/bt/controller/lib  -lbtdm_app  esp-idf/mdns/libmdns.a  esp-idf/console/libconsole.a  platform/esp32/port/gpio_platform/libgpio_platform.a  platform/esp32/port/gpio_platform/libgpio_platform.a  esp-idf/spiffs/libspiffs.a  source/libs/hex_utils/libhex_utils.a  esp-idf/jsmn/libjsmn.a  esp-idf/json/libjson.a  source/libs/crc/libcrc.a  esp-idf/esp_http_client/libesp_http_client.a  esp-idf/tcp_transport/libtcp_transport.a  esp-idf/esp-tls/libesp-tls.a  esp-idf/nghttp/libnghttp.a  dependencies/generic/safestringlib/libsafeclib.a  esp-idf/esp32/libesp32.a  esp-idf/cxx/libcxx.a  esp-idf/newlib/libnewlib.a  esp-idf/freertos/libfreertos.a  esp-idf/heap/libheap.a  esp-idf/log/liblog.a  esp-idf/soc/libsoc.a  esp-idf/esp_rom/libesp_rom.a  esp-idf/esp_common/libesp_common.a  esp-idf/xtensa/libxtensa.a  esp-idf/driver/libdriver.a  esp-idf/esp_event/libesp_event.a  esp-idf/efuse/libefuse.a  esp-idf/app_trace/libapp_trace.a  esp-idf/app_update/libapp_update.a  esp-idf/bootloader_support/libbootloader_support.a  esp-idf/nvs_flash/libnvs_flash.a  esp-idf/pthread/libpthread.a  esp-idf/spi_flash/libspi_flash.a  esp-idf/vfs/libvfs.a  esp-idf/wpa_supplicant/libwpa_supplicant.a  esp-idf/espcoredump/libespcoredump.a  esp-idf/esp_wifi/libesp_wifi.a  esp-idf/esp_ringbuf/libesp_ringbuf.a  esp-idf/tcpip_adapter/libtcpip_adapter.a  esp-idf/esp_eth/libesp_eth.a  esp-idf/lwip/liblwip.a  esp-idf/mbedtls/mbedtls/library/libmbedtls.a  esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a  esp-idf/mbedtls/mbedtls/library/libmbedx509.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcore.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libespnow.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libmesh.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libphy.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libpp.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/librtc.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a  esp-idf/esp32/libesp32.a  esp-idf/cxx/libcxx.a  esp-idf/newlib/libnewlib.a  esp-idf/freertos/libfreertos.a  esp-idf/heap/libheap.a  esp-idf/log/liblog.a  esp-idf/soc/libsoc.a  esp-idf/esp_rom/libesp_rom.a  esp-idf/esp_common/libesp_common.a  esp-idf/xtensa/libxtensa.a  esp-idf/driver/libdriver.a  esp-idf/esp_event/libesp_event.a  esp-idf/efuse/libefuse.a  esp-idf/app_trace/libapp_trace.a  esp-idf/app_update/libapp_update.a  esp-idf/bootloader_support/libbootloader_support.a  esp-idf/nvs_flash/libnvs_flash.a  esp-idf/pthread/libpthread.a  esp-idf/spi_flash/libspi_flash.a  esp-idf/vfs/libvfs.a  esp-idf/wpa_supplicant/libwpa_supplicant.a  esp-idf/espcoredump/libespcoredump.a  esp-idf/esp_wifi/libesp_wifi.a  esp-idf/esp_ringbuf/libesp_ringbuf.a  esp-idf/tcpip_adapter/libtcpip_adapter.a  esp-idf/esp_eth/libesp_eth.a  esp-idf/lwip/liblwip.a  esp-idf/mbedtls/mbedtls/library/libmbedtls.a  esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a  esp-idf/mbedtls/mbedtls/library/libmbedx509.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcore.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libespnow.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libmesh.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libphy.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libpp.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/librtc.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a  -mfix-esp32-psram-cache-issue  -L /build_esp32_app/esp-idf/esp32  -T esp32_out.ld  -u app_main  -L /build_esp32_app/esp-idf/esp32/ld  -T esp32.project.ld  -L /esp-idf/components/esp32/ld  -T esp32.peripherals.ld  -u call_user_start_cpu0  -u ld_include_panic_highint_hdl  -u __cxa_guard_dummy  -lstdc++  esp-idf/pthread/libpthread.a  -u __cxx_fatal_exception  esp-idf/newlib/libnewlib.a  -u newlib_include_locks_impl  -u newlib_include_heap_impl  -u newlib_include_syscalls_impl  -u newlib_include_pthread_impl  -Wl,--undefined=uxTopUsedPriority  -L /esp-idf/components/esp_rom/esp32/ld  -T esp32.rom.ld  -T esp32.rom.libgcc.ld  -T esp32.rom.syscalls.ld  -T esp32.rom.newlib-data.ld  -T esp32.rom.newlib-nano.ld  -Wl,--gc-sections  ../dependencies/espressif/esp-idf/components/xtensa/esp32/libhal.a  -lgcov  -lc_nano  -lm  -lgcc  -u esp_app_desc  -u pthread_include_pthread_impl  -u pthread_include_pthread_cond_impl  -u pthread_include_pthread_local_storage_impl  -u vfs_include_syscalls_impl  -L /esp-idf/components/esp_wifi/lib_esp32
igrr commented 4 years ago

Thanks, this looks correct. Could you please run xtensa-esp32-elf-g++ -mfix-esp32-psram-cache-issue -print-search-dirs and check if the paths under libraries include esp32-psram subdirectories?

Also, if you have switched from esp-2019r2 to esp-2020r1 toolchain, have you removed your CMakeCache.txt? CMake caches the path to the compiler, so it might be possible that the older toolchain is still used.

One other thing, could you please explain what you mean by

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs)

Which exact files have you copied, and to what location?

igrr commented 4 years ago

Oh, i think i see the reason — you are using "nano" C library, which is not compatible with PSRAM. Please check if the issue persists after disabling "nano" option.

sebirdman commented 4 years ago

Removing the nano option seems to have done it, i've successfully ota'd my device from an https endpoint 3 times now. Thanks @igrr

One other thing, could you please explain what you mean by

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs)

I copied xtensa-esp32-elf/lib/esp32-psram files out into the main xtensa-esp32-elf/lib folder, replacing the files there. This least helped me debug that i was doing something wrong and the the toolchain was working.