pauluap
commented
6 years ago I've done some analysis and experiments. I have a bunch of captures and stuff, but right now that's probably too much noise.
Right now, I'm thinking that the issue isn't C++x11 per se, but rather that there's a latent issue that the different code generation output exposed.
I used the logic analyzer to follow the call stack through to lwip_netconn_do_writemore - https://github.com/ARMmbed/mbed-os/blob/a6e27b1b866e7301adcc81f94ea85e0ee7fc63de/features/FEATURE_LWIP/lwip-interface/lwip/src/api/lwip_api_msg.c#L1491.
I didn't check every single call path, but I'm pretty confident that all of them are the same, ultimately ending up at https://github.com/ARMmbed/mbed-os/blob/a6e27b1b866e7301adcc81f94ea85e0ee7fc63de/features/FEATURE_LWIP/lwip-interface/lwip/src/api/lwip_api_msg.c#L1563, calling API_EVENT(conn, NETCONN_EVT_SENDMINUS, 153).
I stopped at that point and then looked at the LWIP debug options. Enabling all the options led to the test application not crashing for a half hour before I stopped it. By comparison, a 'normal' build always ran into problems in less than five minutes. That makes me think that there's a race or resource constraint.
While enabling all debug options led to a stable (presumably) stack, I was able to observe a fault when enabling the debug options one by one. The group of (TCP_DEBUG, TCP_INPUT_DEBUG, TCP_OUTPUT_DEBUG, TCP_RTO_DEBUG, TCP_CWND_DEBUG, TCP_QLEN_DEBUG) was "too stable" for me to capture a failing trace, but I managed to capture a trace from a smaller group of TCP_INPUT_DEBUG and TCP_OUTPUT_DEBUG TCP_QLEN_DEBUG. All of them are attached to this entry as well as the Wireshark capture of the smaller TCP debug group
201803042119_tcp_smaller_group.pcapng.zip
inet_debug.txt
ip_debug.txt
tcp_cwnd_debug.txt
tcp_debug.txt
tcp_input_debug.txt
tcpip_debug.txt
tcp_output_debug.txt
tcp_qlen_debug.txt
tcp_rto_debug.txt
udp_debug.txt
tcp_group.log.zip
tcp_smaller_group.log.zip
I'm hoping for some guidance if I need to do a deep dive. I saw somewhere that there were potential issues around ISR event queues running out of slots. Is there a way for me to extend the various resources involved to check if there's a resource contention issue?
0xc0170
kjbracey
and finally fails enough that the client gives up completely
And it's not like as if one signal got sent to three pins -
The sole pbuf-not-available assert stands out doesn't it?
I believe that the gap in the trace corresponds to packet 24-25
And the end
The TCP state did get screwed too, so seems that we're dropping something somewhere?
But afterwards, it did work till I stopped it.
When I stop and retry, I don't always see the misstep, maybe half of the time.
Target: DISCO_F769NI Toolchain: GCC_ARM 6.3.1 20170215
The background is that I'm working on an application that has a Modbus server component. I'm using modpoll (http://www.modbusdriver.com/modpoll.html) with the command
modpoll <ip-address> -r 1 -c 72 -t 4:hex -l 0to perform torture testing. I had been finding out that the connection fails, then the device isn't responsive anymore, sometimes not even to ping requests.I managed to create a working minimal example at https://github.com/pauluap/mbed-os/tree/lwip_cx11_issue
The twist is that the issue doesn't appear when building with the
-std=gnuc++98(with the commandmbed compile -m DISCO_F769NI -t GCC_ARM) but exhibits problems when building with the-std=c++11option (I did this by runningmbed export -i GCC_ARM -m DISCO_F769NIand then changing the two instances of-std=gnu++98in the generated Makefile)I ended up getting my hands on a logic analyzer. After attaching a logic analyzer, it seems that there may be multiple issues within this problem.
I'm attaching saleae logic analyzer captures as well as wireshark captures. You're more than welcome to download saleae Logic (www.saleae.com/downloads) and Wireshark to open and view the captures. capture_fail_201803041016.zip
One aspect of the problem seems to be that the transmit half of TCP stops working.
In the complete capture screenshot above, it can be observed that the first attach signal leads to a period of more or less normal communication (the solidish gray bars), and then something happens. The client tries a few retries, closes the socket, and tries reopening the socket (after the solidish gray bars, there's spikes without accompanying spike in the Attach signal, that's the retry, then the Attach spikes represent the client closing and trying to reopen the socket).
socket.recvandsocket.send. In this particular screenshot, the communications are (from left to right)8 = recv(&[0], 8)4 = recv(&[8], 4)-3001 = recv(&[0], 8)-3001 = recv(&[0], 8)153 = send(&[0], 153)And the cycle is supposed to repeat ad infinitum. And it does for the gnu++98 build.
In the C++11 build though, things fall off the cliff.
The last socket operation here is a
153 = send(&[0], 153)and then silence.So, the issue seem to tie in with C++11. Why is there even a problem? The LWIP sources are all in C, correct? And that's still built using -std=gnu99
I've been working on developing my codebase all throughout this and other issues, and have a pretty extensive C++11 codebase that I would really love to preserve (aside from having to return to the bad old days of C++98), so I'm willing to help in getting to the bottom of this issue. As you can see, I have a logic analyzer that can assist. The changing data in the Modbus response is worrying though, pointing to some kind of memory address assignment issue within the GCC_ARM toolchain.