search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.2k stars 3.58k forks source link

[Bug] broker jvm crash #21577

Closed zcola closed 9 months ago

zcola commented 11 months ago

Search before asking

Version

2.11.1

Minimal reproduce step

image It happened several times randomly and is not easy to reproduce. There are multiple brokers.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f05ac3c07f2, pid=1, tid=652

hs_err_pid1.log replay_pid1.log docker_broker-2023-11-11.log

What did you expect to see?

broker crash

What did you see instead?

no crash

Anything else?

No response

Are you willing to submit a PR?

Technoboy- commented 11 months ago

Could you provide the dump file ?

lhotari commented 11 months ago

Thanks for reporting. Do you have a chance to upgrade to 2.11.2 ? 2.11.1 runs on 17.0.6 which is rather old. Some JVM crashes were fixed in 17.0.7 .

Btw. Probably not related, but I'd recommend setting /sys/kernel/mm/transparent_hugepage/enabled to madvise when running Java on Linux. (-XX:+UseTransparentHugePages can be used with madvise to opt-in to use THP for the JVM when needed). Alternatively -XX:+AlwaysPreTouch jvm startup option could be used to mitigate some issues that THP causes.

zcola commented 9 months ago

Thanks for reporting. Do you have a chance to upgrade to 2.11.2 ? 2.11.1 runs on 17.0.6 which is rather old. Some JVM crashes were fixed in 17.0.7 .

Btw. Probably not related, but I'd recommend setting /sys/kernel/mm/transparent_hugepage/enabled to madvise when running Java on Linux. (-XX:+UseTransparentHugePages can be used with madvise to opt-in to use THP for the JVM when needed). Alternatively -XX:+AlwaysPreTouch jvm startup option could be used to mitigate some issues that THP causes.

Thank you for your reply. After upgrading to 2.11.2, it has not appeared again for a month.