Setup ATTX instance in cpouta for JY case

[jkesanie]

Description

• Deploy components
• Setup users

DoD

Running instance with access rights to JY people.

Testing

Peer review

[joasilva]

@jkesanie @blankdots We don't have enough free public IP addresses to assign to the new cluster, can I delete your test VMs in Cpouta to free some resources?

[blankdots]

@joasilva yes.

[joasilva]

We need at least 2 more Public Floating IPs for this task, forwarded request to CSC Service Desk.

[joasilva]

Still waiting for the needed additional Public Floating IPs.

[joasilva]

First Kontena stack YML draft here: https://github.com/ATTX-project/platform-deployment/blob/feature-demo-grid/attx-stack-kontena/attx-staging-kontena.yml

Needs some testing and surely some parameter optimisation as well.

[joasilva]

UnifiedViews available at http://vm1196.kaj.pouta.csc.fi:8080/unifiedviews. Please check.

[joasilva]

But attx/es5 doesnt' deploy:

2017-11-13T12:57:29.832Z [attx.es5-1]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)
2017-11-13T12:57:29.832Z [attx.es5-1]: #
2017-11-13T12:57:29.832Z [attx.es5-1]: # There is insufficient memory for the Java Runtime Environment to continue.
2017-11-13T12:57:29.832Z [attx.es5-1]: # Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
2017-11-13T12:57:29.832Z [attx.es5-1]: # Can not save log file, dump to screen..
2017-11-13T12:57:29.832Z [attx.es5-1]: #
2017-11-13T12:57:29.832Z [attx.es5-1]: # There is insufficient memory for the Java Runtime Environment to continue.
2017-11-13T12:57:29.832Z [attx.es5-1]: # Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
2017-11-13T12:57:29.832Z [attx.es5-1]: # Possible reasons:
2017-11-13T12:57:29.832Z [attx.es5-1]: #   The system is out of physical RAM or swap space
2017-11-13T12:57:29.832Z [attx.es5-1]: #   In 32 bit mode, the process size limit was hit
2017-11-13T12:57:29.832Z [attx.es5-1]: # Possible solutions:
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Reduce memory load on the system
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Increase physical memory or swap space
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Check if swap backing store is full
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Use 64 bit Java on a 64 bit OS
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Decrease Java heap size (-Xmx/-Xms)
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Decrease number of Java threads
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Decrease Java thread stack sizes (-Xss)
2017-11-13T12:57:29.832Z [attx.es5-1]: #   Set larger code cache with -XX:ReservedCodeCacheSize=
2017-11-13T12:57:29.832Z [attx.es5-1]: # This output file may be truncated or incomplete.
2017-11-13T12:57:29.832Z [attx.es5-1]: #
2017-11-13T12:57:29.832Z [attx.es5-1]: #  Out of Memory Error (os_linux.cpp:2630), pid=1, tid=0x00007f35d2933700
2017-11-13T12:57:29.832Z [attx.es5-1]: #
2017-11-13T12:57:29.832Z [attx.es5-1]: # JRE version:  (8.0_121-b13) (build )
2017-11-13T12:57:29.832Z [attx.es5-1]: # Java VM: OpenJDK 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
2017-11-13T12:57:29.832Z [attx.es5-1]: # Core dump written. Default location: /usr/share/elasticsearch/core or core.1
2017-11-13T12:57:29.832Z [attx.es5-1]: #
2017-11-13T12:57:29.832Z [attx.es5-1]: 
2017-11-13T12:57:29.832Z [attx.es5-1]: ---------------  T H R E A D  ---------------
2017-11-13T12:57:29.832Z [attx.es5-1]: 
2017-11-13T12:57:29.832Z [attx.es5-1]: Current thread (0x00007f35cc00b800):  JavaThread "Unknown thread" [_thread_in_vm, id=44, stack(0x00007f35d2833000,0x00007f35d2934000)]
2017-11-13T12:57:29.832Z [attx.es5-1]: 
2017-11-13T12:57:29.832Z [attx.es5-1]: Stack: [0x00007f35d2833000,0x00007f35d2934000],  sp=0x00007f35d2932520,  free space=1021k
2017-11-13T12:57:29.832Z [attx.es5-1]: Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
2017-11-13T12:57:29.833Z [attx.es5-1]: V  [libjvm.so+0xa9700d]
2017-11-13T12:57:29.833Z [attx.es5-1]: V  [libjvm.so+0x4ea26c]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x8e2b90]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x8dd27e]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0xa93d2c]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x5f4360]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x4c878d]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x5f51d1]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x5def9f]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0xa5ad19]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0xa5af32]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x6372ef]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0xa3dc3a]
2017-11-13T12:57:29.835Z [attx.es5-1]: V  [libjvm.so+0x6ad815]  JNI_CreateJavaVM+0x65
2017-11-13T12:57:29.835Z [attx.es5-1]: C  [libjli.so+0x2f6e]
2017-11-13T12:57:29.835Z [attx.es5-1]: C  [libjli.so+0x74bd]
2017-11-13T12:57:29.836Z [attx.es5-1]: C  [libpthread.so.0+0x8064]  start_thread+0xc4
2017-11-13T12:57:29.836Z [attx.es5-1]: 
2017-11-13T12:57:29.836Z [attx.es5-1]: 
2017-11-13T12:57:29.836Z [attx.es5-1]: ---------------  P R O C E S S  ---------------
2017-11-13T12:57:29.836Z [attx.es5-1]: 
2017-11-13T12:57:29.836Z [attx.es5-1]: Java Threads: ( => current thread )

[joasilva]

Looking at the docker logs for the failed es5 container on the target node, I could find the following:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)

As per ES VM troubleshooting instructions, I did increase the size of max vm mapped memory, but still it doens't help. Troubleshooting continues.

[joasilva]

Problem solved: resize instances to standard.medium (4G vRAM, 3 vCPUs). attxproject/attx-es5:dev image deployed.

UnifiedViews: http://vm1196.kaj.pouta.csc.fi:8080/unifiedviews/ ElasticSearch: http://vm1190.kaj.pouta.csc.fi:9210