Closed LeeTroy closed 2 years ago
Found the issue, refer to the following code segment: https://github.com/openbmc/bmcweb/blob/ffed87b5ad1797ca966d030e7f97977028d258fa/include/vm_websocket.hpp#L200-L212
While doWrite is an async operation, there are 16 bytes to-be-consumed data in the inputBuffer but the next message (128K+16Byte) coming before the inputBuffer finished writing. Then the inputBuffer->prepare() generate the buffer overflow exception without proper try/catch block to handle it, causing the bmcweb coredump.
L200: Should check with capacity() - size()
in case there are some unprocessed data.
Enlarge the inputBuffer size might be needed as well.
Not seeing the same issue in AST2500, perhaps AST2600 has dual core and it runs faster with bmcweb + nbd_proxy causing timing issue?
The other problem here is
May 04 02:35:55 evb-ast2600 bmcweb[334]: nbd-proxy: wait: Interrupted system call
Which introduced from: https://github.com/openbmc/bmcweb/blob/ffed87b5ad1797ca966d030e7f97977028d258fa/include/vm_websocket.hpp#L37-L42
The proxy.wait()
were interrupted by it's SIGCHILD, might need to use proxy.wait(ec)
and check the error code for retry.
Both of these suggested patches seem reasonable to me; Can you get gerrit reviews opened for them please? I will review them once they're there.
Sure, I'm still running some overnight stress tests, I'll submit to gerrit when I feel confident.
The input buffer size is trickier, I've seen there are more than 200K unprocess data in inputBuffer when BMC is under heavy loaded. However, it might not happen in normal use cases.
Have your patches been submitted?
Ping again for patch submission.
We're seeing the same issue with Nuvoton Poleg BMC. If I increase the input buffer size sufficiently, then I don't see the original issue but websocket read fails. Here's a section of debug output I get at the time of failure, this is with build based on openbmc version 2.9.
Jul 12 22:18:30 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":77] inputBuffer empty. Bailing out Jul 12 22:18:30 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":86] Wrote 65536bytes Jul 12 22:18:30 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":86] Wrote 65536bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":86] Wrote 16bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":77] inputBuffer empty. Bailing out Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":86] Wrote 65536bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:30) [DEBUG "vm_websocket.hpp":86] Wrote 65536bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:31) [DEBUG "vm_websocket.hpp":86] Wrote 16bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:31) [DEBUG "vm_websocket.hpp":77] inputBuffer empty. Bailing out Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:31) [ERROR "websocket.hpp":210] doRead error asio.ssl:336130329 Jul 12 22:18:31 R650xs kernel: unregistered gadget driver 'configfs-gadget' Jul 12 22:18:31 R650xs kernel: block nbd0: shutting down sockets Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:31) [DEBUG "vm_websocket.hpp":117] Read done. Read 0 bytes Jul 12 22:18:31 R650xs bmcweb[933]: (2021-07-12 22:18:31) [ERROR "vm_websocket.hpp":121] Couldn't read from VM port: asio.misc:2
Does anyone have interest in getting the above patches submitted?
Oops, I forgot about this after I applied a patch internally, sorry.
As @singhscp mentioned, I never came out a reasonable value for the buffer, it always failed in some way (especially on multi-core soc). I increase the size to (128K+16Byte)*4, is that okay?
I had also made the buffer 4 times the original size and I didn't see the buffer overrun issue after that.
Submitted: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/45475
Please help to review, thanks!
The aforementioned review is merged. Please reopen if there are any new issues.
Describe the bug Stress testing virtual media over websocket, bmcweb got coredump. We're still investigating if this is a hardware issue, so I post here to check anyone meet this issue before.
Environment We're running these test with both Aspeed AST2600-A1/A2 evaluation board. OpenBMC revision is
01eac59fbe806a32ab857dbceb69b8ede949e724
To Reproduce
fdisk -l /dev/nbd0
dd if=/dev/nbd0 of=/dev/null bs=10M count=1K
, repeatedly until it failed.Is this a regression Normally, the virtual media works fine. This issue only happens under stress dumping data from nbd device.
With bmcweb-logging enabled: