search

Quuxplusone / LLVMBugzillaTest

0 stars 0 forks source link

LLVM ERROR: out of memory for very large .c files #51133

Open Quuxplusone opened 2 years ago

Quuxplusone commented 2 years ago
Bugzilla Link PR52166
Status NEW
Importance P release blocker
Reported by Andrei Roventa (andrei.roventa@windriver.com)
Reported on 2021-10-13 09:07:23 -0700
Last modified on 2021-10-15 13:56:36 -0700
Version 10.0
Hardware PC Linux
CC blitzrakete@gmail.com, dgregor@apple.com, efriedma@quicinc.com, erik.pilkington@gmail.com, llvm-bugs@lists.llvm.org, richard-llvm@metafoo.co.uk
Fixed by commit(s)
Attachments stack_dump_out_of_memory.zip (4472 bytes, application/x-zip-compressed)
Blocks
Blocked by
See also 
Created attachment 25360
stack dump

The following failure is seen frequently (but cannot be reproduced
consistently) for some large files: "LLVM ERROR: out of memory".

The build machine has over 32GB or RAM and most of the time it shows that more
than 20GB are free when these failures occur.

These large files where such failure occur, each has a single very large array
used in testing of the form:

unsigned char pSource13[] =
{ 0x50, 0x4e, 0x6b, 0x57, 0x30, 0x71, 0x33, 0x59, 0x49, 0x31, 0x30, 0x30, ....
0x68, 0x36, 0x71, 0x55, 0x31, 0x66, 0x48, 0x54, 0x51, 0x58, 0x30, 0x3e, 0x55,
0x56, 0x53, 0x47 };

Attached are some of the trace files for this. I cannot attach the source file
as it is extremely large (more than 150MB), but a trimmed down version was
added for reference.

LLVM ERROR: out of memory
Stack dump:
0.  Program arguments:
/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/compilers/llvm-
10.0.1.1/LINUX64/bin/clang -fno-builtin --target=arm64 -ffixed-x18 -mfix-cortex-
a53-835769 -std=c11 -D__vxworks -D__VXWORKS__ -D__ELF__ -D_HAVE_TOOL_XTORS -
nostdlibinc -nostdinc++ -ftls-model=local-exec -fno-builtin -fno-strict-
aliasing -D_USE_INIT_ARRAY -finline-hint-functions -mllvm -inline-threshold=0 -
fno-exceptions -D_NO_EX -fno-rtti -D_NO_RTTI -mno-implicit-float -g -O2 -Wall -
Wconversion -Wno-sign-conversion -DCPU=_VX_ARMARCH8A -DTOOL_FAMILY=llvm -
DTOOL=llvm -D_WRS_KERNEL -DARMEL -DINET -U__USER_LABEL_PREFIX__ -
D__USER_LABEL_PREFIX__= -DARMEL -DINET -U__USER_LABEL_PREFIX__ -
D__USER_LABEL_PREFIX__= -D_WRS_LIB_BUILD -
D_VSB_CONFIG_FILE="/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/h/config/vsbConfig.h"
-
isystem/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/krnl/h/public
-
isystem/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/krnl/h/system
-
I/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/share/h
-
D_VSB_PUBLIC_HDR_DIR=/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/krnl/h/public
-I. -MD -c -o
/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/samples/prebuilt_projects/vsb_ARMARCH8A_GOS_SAFETY/krnl/ARMARCH8A/common/objvxcert_test/zlib_25MB_unique_srting_TC19_in.o
zlib_25MB_unique_srting_TC19_in.c
1.  zlib_25MB_unique_srting_TC19_in.c:568001:49: current parser token ','
clang: error: clang frontend command failed due to signal (use -v to see
invocation)
clang version 10.0.1.1 (http://gitlab.devstar.cloud/compilers/llvm/clang.git
5449acbaec3d4a4a4cf80bc0bbfd17270fce7bcb) (llvm/llvm.git
93ac1237629b578d24c2aedbbe345c20a2cb7767)
Target: arm64
Thread model: posix
InstalledDir:
/home/aroventa/36607/WindRiver_HVP_SC0630ear4/helix/compilers/llvm-10.0.1.1/LINUX64/bin
clang: note: diagnostic msg: PLEASE submit a bug report to
https://bugs.llvm.org/ and include the crash backtrace, preprocessed source,
and associated run script.
clang: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/zlib_25MB_unique_srting_TC19_in-29a3fb.c
clang: note: diagnostic msg: /tmp/zlib_25MB_unique_srting_TC19_in-29a3fb.sh
clang: note: diagnostic msg:
Quuxplusone commented 2 years ago

Attached stack_dump_out_of_memory.zip (4472 bytes, application/x-zip-compressed): stack dump

Quuxplusone commented 2 years ago

"out of memory" means that malloc failed. If you're using a 64-bit Linux clang binary, usually you'll trigger the OOM killer well before you actually hit a malloc failure, so you have some sort of unusual configuration. Maybe ulimit? In any case, you'll probably have to investigate yourself.

Note that files of that form use much more memory than you might expect at first glance because clang constructs multiple AST nodes for each byte. If you control the process generating these files, you might consider doing something else; for example, a string literal uses much less memory.

Quuxplusone commented 2 years ago 
The ulimit look to be quite permissive, at least the important ones (if I am
not mistaking) are unlimited.
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127341
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 127341
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

For the recommended "string literal" value, should having a string like static
char buff[NSIZE] = "\x7a\x6c\x69\x62\x20\x69..." be much less resource
intensive?
Quuxplusone commented 2 years ago 
(In reply to Andrei Roventa from comment #2)
> For the recommended "string literal" value, should having a string like
> static char buff[NSIZE] = "\x7a\x6c\x69\x62\x20\x69..." be much less
> resource intensive?

Yes.  I think last time someone looked, the memory usage for a string like that
ends up at roughly 3x the size of the array?  Vs. your original testcase, where
it's on the order of 100x.
Quuxplusone commented 2 years ago

Oh, and before you go too deep, probably worth confirming you clang binary is actually 64-bit ("file clang" or something like that).