sfc-gh-jyin
commented
2 months ago One quick update:
I was doing another comparison on r7gd.4xlarge instances. The lscpu command returns different results on different kernel versions:
The one has the issue (Kernel version: 5.4.181-99.354):
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Model: 1
BogoMIPS: 2100.00
L1d cache: 64K
L1i cache: 64K
L2 cache: 1024K
L3 cache: 32768K
NUMA node0 CPU(s): 0-15
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp
The one does not have issue (Kernel version: 5.10.215-203.850):
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Model: 1
BogoMIPS: 2100.00
L1d cache: 64K
L1i cache: 64K
L2 cache: 1024K
L3 cache: 32768K
NUMA node0 CPU(s): 0-15
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rngThe node that works fine have those additional feature flags: svei8mm svebf16 i8mm bf16 dgh rng. It might be because of the kernel version difference.
avagin
Description
Hello,
We are currently working on a benchmark to compare gVisor performance on AWS c6gd instances vs c7gd instances. However, when we are running the same workload, our application works fine in c6gd instance, but failed in c7gd instance.
Basically the issue manifests in two ways on c7gd:
1. Getting file status of a bind-mounted file, log shows it succeeds, but client side shows it is not:
On a bad occurrence, the log looks like following:
On a good occurrence, the log looks like following:
As we can see from above, the
fstatatsyscall succeeded in both good and bad occurrence for/usr/lib/python3.8/lib-dynloaddirectory, however somehow user space is not receiving correct return value, and throws error showing module not found.2. Allocating memory for new stack, log showing it succeeds, but caller does not think so:
It is failing at https://github.com/bminor/glibc/blob/master/nptl/allocatestack.c#L380, which is pretty clear that user space receives result of above
mmapas0,instead0xfb57ab76a000as what sentry suggests. Not sure if the shared memory region is corrupted or overridden by other writer.In both cases, we are seeing discrepancy between what gVisor sentry sends back and what user space receives.
3. Getting unhandled user fault right after mmap:
Followings are some investigations we have done so far:
Scalable Vector Extension, but it is unclear how this is related to systrap, as systrap uses general purpose registers to store the syscall result.fstatatsyscalls made on bind-mount files prior to this failing point, and they all work fine.ptraceSteps to reproduce
We are still working on a simple program to reproduce this issue. But in the meantime please let us know if there is any obvious cause to this issue. Thanks!
runsc version
docker version (if using docker)
No response
uname
5.4.181-99.354
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
No response