search

mmisw / orr

ORR - Ontology Registry & Repository integrated system
Apache License 2.0
13 stars 5 forks source link

JVM not getting enough memory in the container #8

Open carueda opened 4 years ago

carueda commented 4 years ago

@brandonnodnarb thanks again for the testing and exposing this issue.

@lewismc / @graybeal FYI

As indicated on slack, I suspect the issue is not enough memory given to the JVM, even though the container itself has plenty.

Below are some snippets from the log and some ctop screenshots.


2020-09-02 00:58:18.094 [http-nio-8080-exec-3] DEBUG org.mmisw.orr.ont.app.OntController -
createOntology:
 user:           AuthUser(brandon)
 uri:            http://synaptica.net/owl
 originalUri:    None
 name:           NASA STI Thesaurus
 version:        20200902T005818
 versionVisibility: Some(owner)
 versionStatus:  Some(unstable)
 log:            None
 date:           2020-09-02T00:58:18
 ownerName:      ~brandon
 ontFileWriter.format: ttl

2020-09-02 00:58:26.958 [http-nio-8080-exec-3] DEBUG org.mmisw.orr.ont.service.OntService - createOntology: md=Map(http://www.w3.org/2000/01/rdf-schema#comment -> List(Synaptica output using OWL syntax.), http://www.w3.org/2002/07/owl#versionInfo -> List(Synaptica OWL), http://www.w3.org/1999/02/22-rdf-syntax-ns#type -> List(http://www.w3.org/2002/07/owl#Ontology))  authorOpt=Some(brandon whitehead)

02-Sep-2020 01:07:05.050 WARNING [cluster-1-thread-1] com.mongodb.ServerStateNotifier.run Exception in monitor thread during notification of server description state change
    java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "AsyncFileHandlerWriter-1259475182" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-nio-8080-ClientPoller-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-nio-8080-ClientPoller-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
02-Sep-2020 01:10:47.569 SEVERE [http-nio-8080-exec-9] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
    java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-nio-8080-exec-7" java.lang.OutOfMemoryError: GC overhead limit exceeded
02-Sep-2020 01:11:38.271 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run Unexpected death of background thread [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
    java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded

image

During the registration processing:

image

carueda commented 4 years ago

There're several references out there related with JVM resource use within containers ... I explored some quite a while ago, and one recent finding now that may be useful is this.