search

GluuFederation / community-edition-setup

Scripts and templates to automate deployment and configuration of the Gluu Server Community Edition
https://gluu.org/docs/ce
MIT License
188 stars 58 forks source link

7 days Long running test failed on 4.3.0 server. (OutOfMemoryError: Java heap space ) #751

Open MohitMaliFtechiz opened 3 years ago

MohitMaliFtechiz commented 3 years ago

Describe.

7 days Long running test failed on 4.3.0 server.

Server breaks after running 2 to 3 days. Able to replicate it on 8 gb with load and 4gb without load system.

logs

11/Aug/2021:11:51:57 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 11 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space ()
[11/Aug/2021:18:02:13 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space ()

Config

Gluu-version 4.3.0 OS:- Ubuntu 20.04

devrimyatar commented 3 years ago

@nynymike @yuriyz @yurem I need guide on how to this problem.

MohitMaliFtechiz commented 3 years ago

hi @mbaser ,

I have placed 4 gb system with 8k user enteries with 4k user login, the system crashes within 2 hours.

MohitMaliFtechiz commented 3 years ago

hi @nynymike ,

I turn the audit logs but after turning it off i am not seeing anylogs on opendj logs.

this is how i turn it off.

go to /opt/opendj/bin
run dsconfig 
Specify OpenDJ LDAP connection parameters
select log publisher
select view and edit an existing  log publishers
select File-Based Audit Logger
select enabled to false
yuriyz commented 3 years ago

@MohitMaliFtechiz does it crash with disabled audit logger ? From message in ticket description it looks like some big data chunk goes to logger which it was not able to process.

yuriyz commented 3 years ago

@MohitMaliFtechiz To circle it back, lets do following : 1) enable logging back 2) add option to create dump file on OOM (e.g. -Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/home/user/dump") 3) Send dump file to me.