Open blookot opened 4 years ago
@blookot
The stack overflow problem can be solved by increasing -Xss
.
We have made improvements to the pipeline compilation which improves single-pipeline compilation/initialization time in 7.5.2 but also introduced a regression for the multi-pipeline use-case (so if using multi-pipelines, do not update to 7.5.2).
The compilation/initialization improvements and the fix for the regression will be in 7.6 (and 7.5.3 if there is a 7.5.3 in the future).
Hi @colinsurprenant
Thank you very much for your answer.
I tested this file again on 7.6.0 that was just released and I'm getting the same compilation error (after 15mn or so) with -Xss28g
...
I attached the output to this comment.
output.txt
I noticed that during the compilation, mostly 1 CPU is used, see top -H output below:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2601 root 20 0 32.122g 3.216g 20372 R 99.7 10.9 3:44.35 Converge Pipeli
2594 root 20 0 32.122g 3.216g 20372 S 8.3 10.9 0:24.81 C1 CompilerThre
2593 root 20 0 32.122g 3.216g 20372 S 3.3 10.9 0:29.26 C2 CompilerThre
2602 root 20 0 32.122g 3.216g 20372 S 3.3 10.9 0:01.26 pool-3-thread-1
2591 root 20 0 32.122g 3.216g 20372 S 2.0 10.9 0:30.06 C2 CompilerThre
2592 root 20 0 32.122g 3.216g 20372 S 1.0 10.9 0:28.92 C2 CompilerThre
PS. During the whole 15mn of processing mentioned above, the 28GB of memory allocated to the JVM is not used! So I don't understand why it would result in a StackOverflow error...
vincent_maury@vincent-maury-logstash-testing:~$ free -m
total used free shared buff/cache available
Mem: 30163 3282 26698 10 182 26536
Swap: 0 0 0
@blookot -Xss28g is way too big. Default for this is typically in Kb (1024). You can see what the default is in your environment using
java -XX:+PrintFlagsFinal -version | grep ThreadStackSize
Note that I tried your -Xss28g
and the result is interesting, it sets the stack at 0! 🤔
java -Xss28g -XX:+PrintFlagsFinal -version | grep ThreadStackSize
intx CompilerThreadStackSize = 0 {pd product}
intx ThreadStackSize := 0 {pd product}
intx VMThreadStackSize = 1024
Sorry @colinsurprenant I didn't explicitely write the settings.
I used -Xms28g
and -Xmx28g
which might be a terrible choice as well...
@blookot
-Xss
I would suggest trying at -Xss2m
and doubling up if you still get a StackOverflowError
. You really should never need more that a 4-8m stack. Thanks again @colinsurprenant I'll try these settings. I initially set the 28GB because of reading our doc page that says: "Leave at least 1GB free for the OS and other processes" ;-)
@blookot leaving just 1Gb as a general rule feels wrong and does not take into account the total physical ram. I will followup with a doc issue to correct that, not sure where this is coming from. Note that ES docs still recommends not more that 50% as a general rule; I would say up to 75% should be ok IMO, especially as we get into high physical memory capacity.
created issue #11605
I'm using Logstash 7.5.0 on GCP Debian 9 (30GB RAM, 28 allocated to JVM).
When running Logstash on a large pipeline (see attached), I'm getting
An exception happened when converging configuration
, see full output below:logstash-v20_oraclemsg.txt