COR is down

[carueda]

COR is down. The AWS instance is up - could others scope.

[carueda]

Problem was that the ORR ran out of memory:

Last few lines of the ORR container:

[cor-admin1@ip-172-30-0-37 ~]$ docker logs -f --tail=40 orr
...
2019-09-28 18:59:08.720 [http-nio-8080-exec-14] DEBUG org.mmisw.orr.ont.app.OntController - raw Accept: Some(*/*)
2019-09-28 19:45:38.846 [http-nio-8080-exec-14] DEBUG org.mmisw.orr.ont.app.OntController - acceptHeader: List(*/*)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/tomcat/hs_err_pid1.log
# [ timer expired, abort... ]
[thread 25 also had an error]

Then:

[cor-admin1@ip-172-30-0-37 COR]$ docker-compose stop orr
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose rm orr
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose up -d

The ORR is up now, but then saw exceptions in the log related with the triple store so decided to restart the whole set of containers:

[cor-admin1@ip-172-30-0-37 COR]$ docker-compose down
[cor-admin1@ip-172-30-0-37 COR]$ docker-compose up -d

Now, all looks good in the log.

Finally, reloaded the triple store by using the Admin option for this in the GUI.

[carueda]

Checking memory settings for the ORR container, it does seem like the max heap size is too small:

[cor-admin1@ip-172-30-0-37 ~]$ docker exec -it orr java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 988.00M
    Using VM: OpenJDK 64-Bit Server VM

openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment 18.9 (build 11.0.4+11)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.4+11, mixed mode)

Some references:

• https://medium.com/adorsys/usecontainersupport-to-the-rescue-e77d6cfea712
• https://developers.redhat.com/blog/2017/03/14/java-inside-docker/

[carueda]

[cor-admin1@ip-172-30-0-37 COR]$ docker exec -it orr java -XX:+PrintFlagsFinal -version | grep MaxHeap
    uintx MaxHeapFreeRatio     = 70                      {manageable} {default}
   size_t MaxHeapSize          = 1035993088                 {product} {ergonomic}

[lewismc]

hi @carueda in second thoughts I'm going to close this. I'll create the supervisord documentatiomn elsewhere.