search

CDLUC3 / mrt-doc

Documentation and Information regarding the Merritt repository
8 stars 4 forks source link

[Audit] Increase JVM size for production Audit #440

Closed mreyescdl closed 4 years ago

mreyescdl commented 4 years ago

Audit 01 production needed a "hard" restart (killing process) of the Audit service on 8/21.

Librato shows a steady increase in the commit memory starting on Thurs/ 8/20 and ending upon restart. Maximum commit memory was at 80%

This looks to be a JVM issue where the Audit processing exceeds the allotted JVM size.
Lets increase the JVM for all production Audit servers from 2GB to 3GB. Physical memory size for all Audit hosts is 4GB.

** Time restarts between audit workers to make sure we don't deplete burst credits.

dloy commented 4 years ago

A number of errors occurred most likely from java OutOfMemoryError:

The memory is currently set at 2G. This needs to be reviewed.

Not clear if problem was a result of aging or actual processing use of Memory.

Should note that multiple lock conditions happened that may also have factored into this problem.

elopatin-uc3 commented 4 years ago

We've decided to increase JVM memory to 3G.

mreyescdl commented 4 years ago

All Audit services are now using a 3GB JVM

elopatin-uc3 commented 4 years ago

Thanks for the update @mreyescdl.