dns_clear_cache() causes lockup (IDFGH-13375)

huming2207 commented 1 month ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.2.2

Espressif SoC revision.

ESP32-S3 (QFN56) (revision v0.1)

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

Custom board

Power Supply used.

External 5V

What is the expected behavior?

Calling dns_clear_cache() should just clear the DNS cache and does fresh DNS query from remote DNS server.

What is the actual behavior?

If a dns_clear_cache() is called in another thread (rather than LWIP task), it may cause lockups.

Someone on the forum also spot the same issue, see: https://www.esp32.com/viewtopic.php?t=25239

Steps to reproduce.

Run a WebSocket RPC client on a network interface for a while
When the network interface connection is down somehow (e.g. someone unplug the ethernet), the program stops the WebSocket client, clear the DNS cache with dns_clear_cache() and switch to another network interface
Start the WS client again, now it will lock up when it query the DNS.
Now it may (or sometimes may not) stuck at sending DNS request ID forever, if LWIP debugging log is enabled.

Debug Logs.

D (19448) lwip: dns_enqueue: "some.internal.domain": use DNS entry 0

I (19468) netmgr: Refresh: netif @ idx=0: en1; 0x3fca2b38, conn? yea, priority=2
D (19468) lwip: dns_enqueue: "some.internal.domain": use DNS pcb 0

I (19478) netmgr: Refresh: netif @ idx=1: pp2; 0x3fca28e8, conn? nah, priority=1
D (19478) lwip: dns_send: dns_servers[0] "some.internal.domain": request

W (19498) netmgr: Default NETIF set to 0x3fca2b38 "en1"
D (19498) lwip: sending DNS request ID 57115 for name "some.internal.domain" to server 0

More Information.

Also see: https://www.esp32.com/viewtopic.php?t=25239

Possible workaround is we probably can issue DNS_TABLE_SIZE times of DNS requests by repeatedly calling gethosebyname() with different valid host names for now. Currently DNS_TABLE_SIZE is 4. But this will waste more data, and the IT team definitely isn't happy to see that when they do security auditing!

huming2207 commented 1 month ago

Sorry I was in a rush earlier and I misread that forum post. Even if the dns_clear_cache() run in the LwIP thread it still may cause lockup. I'm also checking if we clear the DNS by setting TTL to 0 would work or not which initially mentioend by Linetkux Wang.

huming2207 commented 1 month ago

I digged in a bit more, it looks like if there's a DNS query ongoing, and the dns_clear_cache() is called or somehow change the DNS cache table item's state to DNS_STATE_UNUSED, after the DNS query comes back, the LwIP stuff will lock up.

huming2207 commented 4 weeks ago

any updates on this? @espressif-abhikroy

espressif-abhikroy commented 4 weeks ago

any updates on this? @espressif-abhikroy

@huming2207 Thank you for bringing this issue to our attention. It seems that the dns_clear_cache(void) function currently performs a simple memset() without executing dns_call_found(i, NULL);. This leads to the dns_table database being erased, along with the stored callback, which in turn causes tcpip_send_msg_wait_sem() to block indefinitely. As a result, functions like gethostbyname() and other netdb APIs will experience blocking.

I am actively working on a fix, and it will be available shortly. Thank you for your patience.

huming2207 commented 3 weeks ago

Hi @espressif-abhikroy

Thanks for the reply!

This leads to the dns_table database being erased, along with the stored callback, which in turn causes tcpip_send_msg_wait_sem() to block indefinitely. As a result, functions like gethostbyname() and other netdb APIs will experience blocking.

That sounds a bit problematic...😅 I'll wait for the fix. Thanks.

Regards, Jackson

huming2207 commented 1 week ago

May I also ask will this fix planned to be backported to IDFv5.3?

espressif / esp-idf