search

chmorgan / libesphttpd

Libesphttpd - web server for ESP8266 / ESP32
Mozilla Public License 2.0
128 stars 45 forks source link

httpd hangs after browser request #41

Closed phatpaul closed 6 years ago

phatpaul commented 6 years ago

I'm developing a product using libesphttpd. I'm serving 5~15 static files (html, js, css) on the browser initial load. I'm also serving a couple of simple cgi functions.

I've got most everything working for a demonstration in a couple days, but I have a critical bug. Please help!

The httpd server seems to hang occasionally for several minutes. It most often happens during an initial load of the static files.

The ESP still responds to PING while the httpd is frozen.

Sometimes, After several minutes (>20min) of hanging, I see error messages on the console like this: (but sometimes it seems to hang indefinitely)

E (6791432) httpd: send buf tried to write 1032 bytes, wrote -1
E (6791432) httpd-freertos: accept failed
accept: Socket is not connected
E (6791482) httpd: send buf tried to write 1032 bytes, wrote -1

Wireshark shows that the /static/plugins/jquery.min.js file was starting to transfer, but never finishes. There are just a bunch of TCP Keep-Alives after the last TCP segment is received.

TCPView shows that there are still 7 TCP connections from firefox to the ESP. All are ESTABLISHED, except one is FIN_WAIT2. (but the connection that seems to have stalled is listed as ESTABLISHED)

I've hooked-up the JTAG debugger and see that it is stuck in a function. I'll post that info shortly.

I've tried many things to isolate the problem, but it persists:

CONFIG_OPTIMIZATION_LEVEL_RELEASE=y CONFIG_STACK_CHECK=y CONFIG_STACK_CHECK_NORM=y CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=240 CONFIG_SPIRAM_BOOT_INIT=y CONFIG_SPIRAM_IGNORE_NOTFOUND=y CONFIG_SPIRAM_USE_MEMMAP= CONFIG_SPIRAM_USE_CAPS_ALLOC= CONFIG_SPIRAM_USE_MALLOC=y CONFIG_SPIRAM_TYPE_ESPPSRAM32=y CONFIG_SPIRAM_SIZE=4194304 CONFIG_SPIRAM_SPEED_40M=y CONFIG_SPIRAM_MEMTEST=y CONFIG_SPIRAM_CACHE_WORKAROUND= CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=16384 CONFIG_WIFI_LWIP_ALLOCATION_FROM_SPIRAM_FIRST=y CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768 CONFIG_SPIRAM_ALLOW_STACK_EXTERNAL_MEMORY=y CONFIG_TASK_WDT_TIMEOUT_S=15 CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=16 CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=16 CONFIG_ESP32_WIFI_STATIC_TX_BUFFER=y CONFIG_ESP32_WIFI_TX_BUFFER_TYPE=0 CONFIG_ESP32_WIFI_STATIC_TX_BUFFER_NUM=16 CONFIG_EMAC_L2_TO_L3_RX_BUF_MODE=y

CONFIG_FREERTOS_THREAD_LOCAL_STORAGE_POINTERS=4 CONFIG_FREERTOS_ISR_STACKSIZE=4096 CONFIG_SUPPORT_STATIC_ALLOCATION=y

CONFIG_ESPHTTPD_STACK_SIZE=8192 CONFIG_ESPHTTPD_SO_REUSEADDR=y CONFIG_ESPHTTPD_SANITIZE_URLS=y #

LWIP

# CONFIG_L2_TO_L3_COPY=y CONFIG_LWIP_MAX_SOCKETS=16 CONFIG_LWIP_IP_REASSEMBLY=y CONFIG_LWIP_STATS=y CONFIG_ESP_GRATUITOUS_ARP=y CONFIG_GARP_TMR_INTERVAL=60 #

UDP

# CONFIG_TCPIP_TASK_STACK_SIZE=4096

phatpaul commented 6 years ago

more info: ESP-IDF branch release/v3.1 custom board with ESP32-WROVER (16M Flash) custom board also has LAN8720 eth phy (similar circuit as ESP32-EVB from Olimex) I also tried it on the official dev kit ESP-WROVER-KIT

I'm running it on ESP-WROVER-KIT to enable JTAG.

When it hangs, I paused the debugger and the esphttpd thread is:

Thread #26 1073471976 (esphttpd) (Suspended : Container)    
    xQueueGenericReceive() at queue.c:1,591 0x4008f5b9  
    sys_arch_sem_wait() at sys_arch.c:183 0x40147b60    
    0x40000001

I hope this helps. Please suggest some more troubleshooting ideas.

phatpaul commented 6 years ago

I disabled WiFi and am just using Ethernet. The server seems a little bit more stable now, but after I refreshed the browser a few times, it hung. Then the ESP reset with this message:

E (242498) httpd: send buf tried to write 1032 bytes, wrote -1
E (242498) httpd: send buf tried to write 1032 bytes, wrote -1
E (242498) httpd: send buf tried to write 1032 bytes, wrote -1
E (242508) httpd: send buf tried to write 1032 bytes, wrote -1
E (242508) httpd: send buf tried to write 1032 bytes, wrote -1
E (242518) httpd: send buf tried to write 1032 bytes, wrote -1
E (242538) httpd: send buf tried to write 138 bytes, wrote -1
E (242548) httpd: send buf tried to write 138 bytes, wrote -1
assertion "tcp_write: no pbufs on queue => both queues empty" failed: file "C:/m
sys32/home/labview/lumitec-dev-fw-plibox-esp32/Tools/esp-idf/components/lwip/cor
e/tcp_out.c", line 350, function: tcp_write_checks
abort() was called at PC 0x400dc10f on core 1
0x400dc10f: __assert_func at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa
-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/asse
rt.c:63 (discriminator 8)

Backtrace: 0x40091030:0x3ffc9f80 0x4009120b:0x3ffc9fa0 0x400dc10f:0x3ffc9fc0 0x4
012faec:0x3ffc9ff0 0x4012ff58:0x3ffca010 0x4012d592:0x3ffca080 0x4012e581:0x3ffc
a0a0 0x4012e895:0x3ffca0c0
0x40091030: invoke_abort at C:/msys32/home/labview/lumitec-dev-fw-plibox-esp32/T
ools/esp-idf/components/esp32/panic.c:649

0x4009120b: abort at C:/msys32/home/labview/lumitec-dev-fw-plibox-esp32/Tools/es
p-idf/components/esp32/panic.c:649

0x400dc10f: __assert_func at /Users/ivan/e/newlib_xtensa-2.2.0-bin/newlib_xtensa
-2.2.0/xtensa-esp32-elf/newlib/libc/stdlib/../../../.././newlib/libc/stdlib/asse
rt.c:63 (discriminator 8)

0x4012faec: tcp_write_checks at C:/msys32/home/labview/lumitec-dev-fw-plibox-esp
32/Tools/esp-idf/components/lwip/core/tcp_out.c:349 (discriminator 3)

0x4012ff58: tcp_write at C:/msys32/home/labview/lumitec-dev-fw-plibox-esp32/Tool
s/esp-idf/components/lwip/core/tcp_out.c:404

0x4012d592: lwip_netconn_do_writemore at C:/msys32/home/labview/lumitec-dev-fw-p
libox-esp32/Tools/esp-idf/components/lwip/api/api_msg.c:1892 (discriminator 6)

0x4012e581: lwip_netconn_do_write at C:/msys32/home/labview/lumitec-dev-fw-plibo
x-esp32/Tools/esp-idf/components/lwip/api/api_msg.c:1892 (discriminator 6)

0x4012e895: tcpip_thread at C:/msys32/home/labview/lumitec-dev-fw-plibox-esp32/T
ools/esp-idf/components/lwip/api/tcpip.c:481

Rebooting...
Si03 commented 6 years ago

@phatpaul I have a very similar setup to you, this library, LAN8270, wifi, and UART, maybe less files per page but a fair bit of json moving about via cgi and it's working fine. I have seen the same error during development though (wrote -1) and can't remember for the time being exactly what it was. Are you using the basic authentication at all? I do know that was a problem in relation to cgi calls. Also had some issues with cross site cgi and authentication because even in a different folder they are deemed cross-site. You have to enable cors via menuconfig. Not a lot of help, sorry. Si.

phatpaul commented 6 years ago

I think I got it!

I had this option not enabled for some reason: CONFIG_SPIRAM_CACHE_WORKAROUND=

Enabled that and no more hang-ups. :-)

Thanks @Si03 for giving me some confidence that this thing can work.

chmorgan commented 6 years ago

@phatpaul I actually mentioned a similar feature some time ago but related to the GPIO pins. https://github.com/espressif/esp-idf/issues/1742

I assume that you've enable SPIRAM but not that particular option. Couldn't hurt to mention this to the esp-idf guys.

phatpaul commented 6 years ago

It's totally my fault. I even remember reading the docs about the SPIRAM workaround:

https://espressif-docs.readthedocs-hosted.com/projects/esp-idf/en/stable/api-guides/external-ram.html#esp32-rev-v1

But I must have figured they had fixed the silicon already. Not true, my project is running on revision 1 silicon. See the output from make flash 

----------------------
Flashing the image ...
----------------------
esptool.py v2.5.0
Serial port COM17
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse
...

Sorry for falsely blaming this excellent httpd software. The SPIRAM hardware bug probably causes plenty of weird and seemingly unrelated malfunctions.