search

nshou / elasticsearch-kibana

Simple and lightweight docker image for previewing Elasticsearch and Kibana.
https://hub.docker.com/r/nshou/elasticsearch-kibana
MIT License
46 stars 24 forks source link

Docker hangs on Status: Downloaded newer image for nshou/elasticsearch-kibana:latest #4

Closed nshou closed 5 years ago

nshou commented 5 years ago

(Original issue 4 created by None on 2018-03-02T14:09:19.053331+00:00)

Running sudo docker run -it --rm -p 9200:9200 -p 5601:5601 --name esk nshou/elasticsearch-kibana causes docker to hang on Status: Downloaded newer image for nshou/elasticsearch-kibana:latest.

Tried other ELK images which work fine.

nshou commented 5 years ago

(Original issue 4 last updated on 2018-03-05T14:20:16.438800+00:00)

(Issue automaticaly closed due to status in Bitbucket: resolved)

nshou commented 5 years ago

(Original comment by nshou on 2018-03-03T05:10:29.279421+00:00)

Please try removing "-it" option.

Elasticsearch and Kibana in that container are indeed successfully running and the output of the docker command is normal even if you specify the "-it" option. However, what the option does is to bind tty in the container with the stdout of your shell and the output of Elasticsearch and Kibana in the container is suppressed by their "quiet" options intentionally, so you might have thought that they appeared to be hung up.

nshou commented 5 years ago

(Original comment by nathandaly on 2018-03-03T10:36:09.686126+00:00)

Tried it on 3 different instances.

They all produce similar results without running the -it option. Strangely on occasion the run command fails and I get API not enough memory error and sometimes the run command just hangs but the container does spin up and all services are available but no elasticsearch?

ef.PNG

nshou commented 5 years ago

(Original comment by nshou on 2018-03-03T17:45:28.308746+00:00)

Nathan,

You are probably mentioning two independent cases. First one is the issue related to memory. It looks to be a common problem in using Elasticsearch, addressed in issue #1, so it is worth for you first trying the measure talked about in issue #1.

The other seems to be the situation where all the processes in the container have started except Elasticsearch process. Starting up Elasticsearch is a CPU-intensive task and takes longer than Kibana. It often occurs that Kibana cannot connect to Elasticsearch that is still under preparation for its bootup.

I was wondering if you could attach the followings here for each environment so that I can investigate more and test my hypotheses pointed out above:

$ free -m
$ docker run -d -p 9200:9200 nshou/elasticsearch-kibana sh elasticsearch/bin/elasticsearch -E http.host=0.0.0.0 -v
### wait for a few minutes ###
$ docker exec ${CONTAINERID} cat /home/elasticsearch/elasticsearch/logs/elasticsearch.log
$ docker exec ${CONTAINERID} cat /home/elasticsearch/elasticsearch/logs/gc.log.0.current

Thank you for your comment.

nshou commented 5 years ago

(Original comment by nathandaly on 2018-03-04T09:56:23.722208+00:00)

Hey,

Firstly thanks for your help on this here are the results.

Digital ocean 2cpu 8gb: digitalocean-1.png digitalocean-2.png

Azure lower spec: azure-1.png

elk@elk-container:~$ docker exec ${CONTAINERID} cat /home/elasticsearch/elasticsearch/logs/elasticsearch.log
Error: No such container: cat
elk@elk-container:~$ docker exec a95690ec98c7eb4b17772f38fc7425c228acc0f8b9a661290c9d95dc33517c54 cat /home/elasticsearch/elasticsearch/logs/elasticsearch.log
[2018-03-04T09:41:06,644][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-04T09:41:07,043][INFO ][o.e.e.NodeEnvironment    ] [pxEJdiU] using [1] data paths, mounts [[/ (overlay)]], net usable_space [25.6gb], net total_space [28.3gb], types [overlay]
[2018-03-04T09:41:07,043][INFO ][o.e.e.NodeEnvironment    ] [pxEJdiU] heap size [1015.6mb], compressed ordinary object pointers [true]
[2018-03-04T09:41:07,047][INFO ][o.e.n.Node               ] node name [pxEJdiU] derived from node ID [pxEJdiUQR42jHwR2QpoycQ]; set [node.name] to override
[2018-03-04T09:41:07,048][INFO ][o.e.n.Node               ] version[6.2.1], pid[1], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-04T09:41:07,048][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/home/elasticsearch/elasticsearch.tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/elasticsearch/elasticsearch, -Des.path.conf=/home/elasticsearch/elasticsearch/config]
[2018-03-04T09:41:10,654][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [aggs-matrix-stats]
[2018-03-04T09:41:10,656][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [analysis-common]
[2018-03-04T09:41:10,660][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [ingest-common]
[2018-03-04T09:41:10,661][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [lang-expression]
[2018-03-04T09:41:10,661][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [lang-mustache]
[2018-03-04T09:41:10,661][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [lang-painless]
[2018-03-04T09:41:10,662][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [mapper-extras]
[2018-03-04T09:41:10,662][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [parent-join]
[2018-03-04T09:41:10,662][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [percolator]
[2018-03-04T09:41:10,662][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [rank-eval]
[2018-03-04T09:41:10,663][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [reindex]
[2018-03-04T09:41:10,663][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [repository-url]
[2018-03-04T09:41:10,665][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [transport-netty4]
[2018-03-04T09:41:10,665][INFO ][o.e.p.PluginsService     ] [pxEJdiU] loaded module [tribe]
[2018-03-04T09:41:10,666][INFO ][o.e.p.PluginsService     ] [pxEJdiU] no plugins loaded
[2018-03-04T09:41:27,266][INFO ][o.e.d.DiscoveryModule    ] [pxEJdiU] using discovery type [zen]
[2018-03-04T09:41:30,916][INFO ][o.e.n.Node               ] initialized
[2018-03-04T09:41:30,917][INFO ][o.e.n.Node               ] [pxEJdiU] starting ...
[2018-03-04T09:41:32,139][INFO ][o.e.t.TransportService   ] [pxEJdiU] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2018-03-04T09:41:32,202][WARN ][o.e.b.BootstrapChecks    ] [pxEJdiU] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-03-04T09:41:35,492][INFO ][o.e.c.s.MasterService    ] [pxEJdiU] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {pxEJdiU}{pxEJdiUQR42jHwR2QpoycQ}{KQXFw2M7QqSl5UBydIV9Ng}{127.0.0.1}{127.0.0.1:9300}
[2018-03-04T09:41:35,541][INFO ][o.e.c.s.ClusterApplierService] [pxEJdiU] new_master {pxEJdiU}{pxEJdiUQR42jHwR2QpoycQ}{KQXFw2M7QqSl5UBydIV9Ng}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {pxEJdiU}{pxEJdiUQR42jHwR2QpoycQ}{KQXFw2M7QqSl5UBydIV9Ng}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-04T09:41:35,642][INFO ][o.e.h.n.Netty4HttpServerTransport] [pxEJdiU] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2018-03-04T09:41:35,645][INFO ][o.e.n.Node               ] [pxEJdiU] started
[2018-03-04T09:41:35,711][INFO ][o.e.g.GatewayService     ] [pxEJdiU] recovered [0] indices into cluster_state
elk@elk-container:~$ docker exec a95690ec98c7eb4b17772f38fc7425c228acc0f8b9a661290c9d95dc33517c54 cat /home/elasticsearch/elasticsearch/logs/gc.log.0.current
OpenJDK 64-Bit Server VM (25.151-b12) for linux-amd64 JRE (1.8.0_151-b12), built on Nov 21 2017 11:32:26 by "buildozer" with gcc 6.4.0
Memory: 4k page, physical 1706596k(213096k free), swap 0k(0k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=87244800 -XX:MaxTenuringThreshold=6 -XX:NewSize=87244800 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=174489600 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2018-03-04T09:41:02.264+0000: 4.212: Total time for which application threads were stopped: 0.0001374 seconds, Stopping threads took: 0.0000320 seconds
2018-03-04T09:41:03.270+0000: 5.218: Total time for which application threads were stopped: 0.0057100 seconds, Stopping threads took: 0.0015602 seconds
2018-03-04T09:41:03.710+0000: 5.658: Total time for which application threads were stopped: 0.0001827 seconds, Stopping threads took: 0.0000292 seconds
2018-03-04T09:41:03.922+0000: 5.871: Total time for which application threads were stopped: 0.0001892 seconds, Stopping threads took: 0.0000304 seconds
2018-03-04T09:41:03.987+0000: 5.936: Total time for which application threads were stopped: 0.0001954 seconds, Stopping threads took: 0.0000285 seconds
2018-03-04T09:41:04.990+0000: 6.939: Total time for which application threads were stopped: 0.0001586 seconds, Stopping threads took: 0.0000429 seconds
2018-03-04T09:41:05.005+0000: 6.954: Total time for which application threads were stopped: 0.0001832 seconds, Stopping threads took: 0.0000220 seconds
2018-03-04T09:41:05.559+0000: 7.507: Total time for which application threads were stopped: 0.0002864 seconds, Stopping threads took: 0.0000303 seconds
2018-03-04T09:41:05.594+0000: 7.542: Total time for which application threads were stopped: 0.0002071 seconds, Stopping threads took: 0.0000157 seconds
2018-03-04T09:41:05.794+0000: 7.742: Total time for which application threads were stopped: 0.0003386 seconds, Stopping threads took: 0.0001605 seconds
2018-03-04T09:41:05.931+0000: 7.879: [GC (Allocation Failure) 2018-03-04T09:41:05.931+0000: 7.880: [ParNew
Desired survivor size 4358144 bytes, new threshold 1 (max 6)
- age   1:    8681984 bytes,    8681984 total
: 68160K->8511K(76672K), 0.0492946 secs] 68160K->9849K(1040064K), 0.0493954 secs] [Times: user=0.05 sys=0.00, real=0.05 secs]
2018-03-04T09:41:05.980+0000: 7.929: Total time for which application threads were stopped: 0.0495761 seconds, Stopping threads took: 0.0000310 seconds
2018-03-04T09:41:06.000+0000: 7.948: Total time for which application threads were stopped: 0.0001021 seconds, Stopping threads took: 0.0000503 seconds
2018-03-04T09:41:06.002+0000: 7.951: Total time for which application threads were stopped: 0.0000570 seconds, Stopping threads took: 0.0000125 seconds
2018-03-04T09:41:06.003+0000: 7.951: Total time for which application threads were stopped: 0.0000450 seconds, Stopping threads took: 0.0000092 seconds
2018-03-04T09:41:06.003+0000: 7.951: Total time for which application threads were stopped: 0.0000417 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:06.003+0000: 7.951: Total time for which application threads were stopped: 0.0000410 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:06.003+0000: 7.951: Total time for which application threads were stopped: 0.0000410 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000422 seconds, Stopping threads took: 0.0000091 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000405 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000406 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000408 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000408 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000407 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.003+0000: 7.952: Total time for which application threads were stopped: 0.0000404 seconds, Stopping threads took: 0.0000079 seconds
2018-03-04T09:41:06.004+0000: 7.952: Total time for which application threads were stopped: 0.0000420 seconds, Stopping threads took: 0.0000088 seconds
2018-03-04T09:41:06.004+0000: 7.952: Total time for which application threads were stopped: 0.0000407 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.004+0000: 7.952: Total time for which application threads were stopped: 0.0000409 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.004+0000: 7.952: Total time for which application threads were stopped: 0.0000408 seconds, Stopping threads took: 0.0000083 seconds
2018-03-04T09:41:06.004+0000: 7.953: Total time for which application threads were stopped: 0.0000407 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:06.004+0000: 7.953: Total time for which application threads were stopped: 0.0000408 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:06.004+0000: 7.953: Total time for which application threads were stopped: 0.0000506 seconds, Stopping threads took: 0.0000083 seconds
2018-03-04T09:41:06.383+0000: 8.332: Total time for which application threads were stopped: 0.0003027 seconds, Stopping threads took: 0.0000312 seconds
2018-03-04T09:41:06.927+0000: 8.875: Total time for which application threads were stopped: 0.0003149 seconds, Stopping threads took: 0.0000285 seconds
2018-03-04T09:41:07.914+0000: 9.863: Total time for which application threads were stopped: 0.0001687 seconds, Stopping threads took: 0.0000338 seconds
2018-03-04T09:41:07.940+0000: 9.889: [GC (Allocation Failure) 2018-03-04T09:41:07.940+0000: 9.889: [ParNew
Desired survivor size 4358144 bytes, new threshold 6 (max 6)
- age   1:    4262144 bytes,    4262144 total
: 76671K->7189K(76672K), 0.0861368 secs] 78009K->11412K(1040064K), 0.0891804 secs] [Times: user=0.09 sys=0.00, real=0.09 secs]
2018-03-04T09:41:08.030+0000: 9.978: Total time for which application threads were stopped: 0.0893539 seconds, Stopping threads took: 0.0000150 seconds
2018-03-04T09:41:08.037+0000: 9.986: Total time for which application threads were stopped: 0.0000741 seconds, Stopping threads took: 0.0000251 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000542 seconds, Stopping threads took: 0.0000119 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000390 seconds, Stopping threads took: 0.0000088 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000377 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000368 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000370 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000378 seconds, Stopping threads took: 0.0000084 seconds
2018-03-04T09:41:08.038+0000: 9.987: Total time for which application threads were stopped: 0.0000387 seconds, Stopping threads took: 0.0000087 seconds
2018-03-04T09:41:08.039+0000: 9.987: Total time for which application threads were stopped: 0.0000365 seconds, Stopping threads took: 0.0000079 seconds
2018-03-04T09:41:08.039+0000: 9.987: Total time for which application threads were stopped: 0.0000367 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.039+0000: 9.987: Total time for which application threads were stopped: 0.0000369 seconds, Stopping threads took: 0.0000081 seconds
2018-03-04T09:41:08.039+0000: 9.987: Total time for which application threads were stopped: 0.0000368 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.039+0000: 9.987: Total time for which application threads were stopped: 0.0000365 seconds, Stopping threads took: 0.0000079 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000380 seconds, Stopping threads took: 0.0000085 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000366 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000366 seconds, Stopping threads took: 0.0000079 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000361 seconds, Stopping threads took: 0.0000078 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000368 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000363 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.039+0000: 9.988: Total time for which application threads were stopped: 0.0000535 seconds, Stopping threads took: 0.0000083 seconds
2018-03-04T09:41:08.041+0000: 9.989: Total time for which application threads were stopped: 0.0000524 seconds, Stopping threads took: 0.0000112 seconds
2018-03-04T09:41:08.041+0000: 9.989: Total time for which application threads were stopped: 0.0000435 seconds, Stopping threads took: 0.0000084 seconds
2018-03-04T09:41:08.041+0000: 9.990: Total time for which application threads were stopped: 0.0000364 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.041+0000: 9.990: Total time for which application threads were stopped: 0.0000363 seconds, Stopping threads took: 0.0000078 seconds
2018-03-04T09:41:08.041+0000: 9.990: Total time for which application threads were stopped: 0.0000363 seconds, Stopping threads took: 0.0000080 seconds
2018-03-04T09:41:08.041+0000: 9.990: Total time for which application threads were stopped: 0.0000363 seconds, Stopping threads took: 0.0000079 seconds
2018-03-04T09:41:08.041+0000: 9.990: Total time for which application threads were stopped: 0.0000370 seconds, Stopping threads took: 0.0000084 seconds
2018-03-04T09:41:08.803+0000: 10.752: [GC (Allocation Failure) 2018-03-04T09:41:08.803+0000: 10.752: [ParNew
Desired survivor size 4358144 bytes, new threshold 1 (max 6)
- age   1:    4537080 bytes,    4537080 total
- age   2:     912104 bytes,    5449184 total
: 75349K->5800K(76672K), 0.0473556 secs] 79572K->10024K(1040064K), 0.0474567 secs] [Times: user=0.04 sys=0.00, real=0.05 secs]
2018-03-04T09:41:08.851+0000: 10.799: Total time for which application threads were stopped: 0.0478779 seconds, Stopping threads took: 0.0000376 seconds
2018-03-04T09:41:09.357+0000: 11.305: Total time for which application threads were stopped: 0.0001101 seconds, Stopping threads took: 0.0000319 seconds
2018-03-04T09:41:09.366+0000: 11.315: Total time for which application threads were stopped: 0.0000793 seconds, Stopping threads took: 0.0000234 seconds
2018-03-04T09:41:09.376+0000: 11.324: Total time for which application threads were stopped: 0.0000989 seconds, Stopping threads took: 0.0000301 seconds
2018-03-04T09:41:09.502+0000: 11.451: Total time for which application threads were stopped: 0.0001222 seconds, Stopping threads took: 0.0000305 seconds
2018-03-04T09:41:09.504+0000: 11.453: Total time for which application threads were stopped: 0.0000771 seconds, Stopping threads took: 0.0000221 seconds
2018-03-04T09:41:09.619+0000: 11.568: [GC (Allocation Failure) 2018-03-04T09:41:09.620+0000: 11.568: [ParNew
Desired survivor size 4358144 bytes, new threshold 6 (max 6)
- age   1:    3887040 bytes,    3887040 total
: 73960K->4882K(76672K), 0.0650936 secs] 78184K->10140K(1040064K), 0.0651988 secs] [Times: user=0.07 sys=0.00, real=0.06 secs]
2018-03-04T09:41:09.685+0000: 11.633: Total time for which application threads were stopped: 0.0653836 seconds, Stopping threads took: 0.0000398 seconds
2018-03-04T09:41:09.733+0000: 11.682: Total time for which application threads were stopped: 0.0000993 seconds, Stopping threads took: 0.0000284 seconds
2018-03-04T09:41:09.750+0000: 11.698: Total time for which application threads were stopped: 0.0000803 seconds, Stopping threads took: 0.0000228 seconds
2018-03-04T09:41:09.904+0000: 11.853: Total time for which application threads were stopped: 0.0001236 seconds, Stopping threads took: 0.0000437 seconds
2018-03-04T09:41:09.927+0000: 11.876: Total time for which application threads were stopped: 0.0001030 seconds, Stopping threads took: 0.0000320 seconds
2018-03-04T09:41:10.053+0000: 12.002: Total time for which application threads were stopped: 0.0001105 seconds, Stopping threads took: 0.0000324 seconds
2018-03-04T09:41:10.071+0000: 12.019: Total time for which application threads were stopped: 0.0040799 seconds, Stopping threads took: 0.0000239 seconds
2018-03-04T09:41:10.197+0000: 12.146: Total time for which application threads were stopped: 0.0001371 seconds, Stopping threads took: 0.0000356 seconds
2018-03-04T09:41:10.217+0000: 12.166: Total time for which application threads were stopped: 0.0000836 seconds, Stopping threads took: 0.0000232 seconds
2018-03-04T09:41:10.227+0000: 12.176: Total time for which application threads were stopped: 0.0001107 seconds, Stopping threads took: 0.0000238 seconds
2018-03-04T09:41:10.356+0000: 12.305: [GC (Allocation Failure) 2018-03-04T09:41:10.356+0000: 12.305: [ParNew
Desired survivor size 4358144 bytes, new threshold 6 (max 6)
- age   1:    2956792 bytes,    2956792 total
- age   2:     277272 bytes,    3234064 total
: 73042K->4053K(76672K), 0.0949284 secs] 78300K->9311K(1040064K), 0.0950325 secs] [Times: user=0.09 sys=0.00, real=0.10 secs]
2018-03-04T09:41:10.452+0000: 12.400: Total time for which application threads were stopped: 0.0952118 seconds, Stopping threads took: 0.0000311 seconds
2018-03-04T09:41:10.565+0000: 12.523: Total time for which application threads were stopped: 0.0098027 seconds, Stopping threads took: 0.0000306 seconds
2018-03-04T09:41:10.588+0000: 12.537: Total time for which application threads were stopped: 0.0000853 seconds, Stopping threads took: 0.0000266 seconds
2018-03-04T09:41:11.589+0000: 13.537: Total time for which application threads were stopped: 0.0001665 seconds, Stopping threads took: 0.0000578 seconds
2018-03-04T09:41:11.600+0000: 13.549: Total time for which application threads were stopped: 0.0001452 seconds, Stopping threads took: 0.0000251 seconds
2018-03-04T09:41:11.708+0000: 13.657: Total time for which application threads were stopped: 0.0002275 seconds, Stopping threads took: 0.0000347 seconds
2018-03-04T09:41:11.740+0000: 13.689: Total time for which application threads were stopped: 0.0003086 seconds, Stopping threads took: 0.0000142 seconds
2018-03-04T09:41:12.227+0000: 14.175: Total time for which application threads were stopped: 0.0002286 seconds, Stopping threads took: 0.0000219 seconds
2018-03-04T09:41:12.384+0000: 14.333: Total time for which application threads were stopped: 0.0002455 seconds, Stopping threads took: 0.0000360 seconds
2018-03-04T09:41:12.456+0000: 14.404: [GC (CMS Initial Mark) [1 CMS-initial-mark: 5257K(963392K)] 54124K(1040064K), 0.1271363 secs] [Times: user=0.13 sys=0.00, real=0.13 secs]
2018-03-04T09:41:12.583+0000: 14.531: Total time for which application threads were stopped: 0.1272878 seconds, Stopping threads took: 0.0000405 seconds
2018-03-04T09:41:12.592+0000: 14.541: [CMS-concurrent-mark-start]
2018-03-04T09:41:12.661+0000: 14.610: [CMS-concurrent-mark: 0.069/0.069 secs] [Times: user=0.06 sys=0.00, real=0.07 secs]
2018-03-04T09:41:12.663+0000: 14.611: Total time for which application threads were stopped: 0.0001908 seconds, Stopping threads took: 0.0000193 seconds
2018-03-04T09:41:12.667+0000: 14.616: [CMS-concurrent-preclean-start]
2018-03-04T09:41:12.686+0000: 14.635: [CMS-concurrent-preclean: 0.019/0.019 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-03-04T09:41:12.686+0000: 14.635: [CMS-concurrent-abortable-preclean-start]
2018-03-04T09:41:12.754+0000: 14.703: Total time for which application threads were stopped: 0.0022206 seconds, Stopping threads took: 0.0000196 seconds
2018-03-04T09:41:12.978+0000: 14.926: Total time for which application threads were stopped: 0.0001796 seconds, Stopping threads took: 0.0000215 seconds
2018-03-04T09:41:13.051+0000: 15.000: Total time for which application threads were stopped: 0.0002243 seconds, Stopping threads took: 0.0000191 seconds
2018-03-04T09:41:13.251+0000: 15.200: Total time for which application threads were stopped: 0.0001050 seconds, Stopping threads took: 0.0000224 seconds
2018-03-04T09:41:13.988+0000: 15.936: Total time for which application threads were stopped: 0.0002388 seconds, Stopping threads took: 0.0000361 seconds
2018-03-04T09:41:14.490+0000: 16.439: [GC (Allocation Failure) 2018-03-04T09:41:14.490+0000: 16.439: [ParNew
Desired survivor size 4358144 bytes, new threshold 1 (max 6)
- age   1:    7527928 bytes,    7527928 total
- age   2:     171552 bytes,    7699480 total
- age   3:     272152 bytes,    7971632 total
: 72213K->8512K(76672K), 0.1964040 secs] 77471K->16391K(1040064K), 0.1965383 secs] [Times: user=0.19 sys=0.00, real=0.20 secs]
2018-03-04T09:41:14.687+0000: 16.636: Total time for which application threads were stopped: 0.1967389 seconds, Stopping threads took: 0.0000209 seconds
2018-03-04T09:41:14.715+0000: 16.663: Total time for which application threads were stopped: 0.0082227 seconds, Stopping threads took: 0.0081571 seconds
2018-03-04T09:41:14.715+0000: 16.663: Total time for which application threads were stopped: 0.0000489 seconds, Stopping threads took: 0.0000098 seconds
2018-03-04T09:41:14.715+0000: 16.663: Total time for which application threads were stopped: 0.0000399 seconds, Stopping threads took: 0.0000078 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000438 seconds, Stopping threads took: 0.0000085 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000428 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000406 seconds, Stopping threads took: 0.0000076 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000427 seconds, Stopping threads took: 0.0000082 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000403 seconds, Stopping threads took: 0.0000077 seconds
2018-03-04T09:41:14.715+0000: 16.664: Total time for which application threads were stopped: 0.0000406 seconds, Stopping threads took: 0.0000075 seconds
2018-03-04T09:41:15.727+0000: 17.675: Total time for which application threads were stopped: 0.0001221 seconds, Stopping threads took: 0.0000539 seconds
2018-03-04T09:41:16.593+0000: 18.542: [CMS-concurrent-abortable-preclean: 2.777/3.907 secs] [Times: user=3.82 sys=0.04, real=3.91 secs]
2018-03-04T09:41:16.599+0000: 18.548: [GC (CMS Final Remark) [YG occupancy: 58770 K (76672 K)]2018-03-04T09:41:16.599+0000: 18.548: [Rescan (parallel) , 0.1384582 secs]2018-03-04T09:41:16.738+0000: 18.686: [weak refs processing, 0.0003744 secs]2018-03-04T09:41:16.738+0000: 18.687: [class unloading, 0.0087883 secs]2018-03-04T09:41:16.747+0000: 18.695: [scrub symbol table, 0.0086847 secs]2018-03-04T09:41:16.756+0000: 18.704: [scrub string table, 0.0006151 secs][1 CMS-remark: 7879K(963392K)] 66649K(1040064K), 0.1578434 secs] [Times: user=0.16 sys=0.00, real=0.15 secs]
2018-03-04T09:41:16.757+0000: 18.706: Total time for which application threads were stopped: 0.1636595 seconds, Stopping threads took: 0.0057133 seconds
2018-03-04T09:41:16.774+0000: 18.723: [CMS-concurrent-sweep-start]
2018-03-04T09:41:16.778+0000: 18.726: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-03-04T09:41:16.778+0000: 18.726: [CMS-concurrent-reset-start]
2018-03-04T09:41:16.794+0000: 18.743: [CMS-concurrent-reset: 0.016/0.016 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2018-03-04T09:41:17.292+0000: 19.241: [GC (Allocation Failure) 2018-03-04T09:41:17.293+0000: 19.241: [ParNew
Desired survivor size 4358144 bytes, new threshold 6 (max 6)
- age   1:    3704808 bytes,    3704808 total
: 76672K->5682K(76672K), 0.2075888 secs] 83203K->20159K(1040064K), 0.2082567 secs] [Times: user=0.21 sys=0.00, real=0.21 secs]
2018-03-04T09:41:17.500+0000: 19.449: Total time for which application threads were stopped: 0.2084476 seconds, Stopping threads took: 0.0000352 seconds
2018-03-04T09:41:18.522+0000: 20.470: Total time for which application threads were stopped: 0.0110741 seconds, Stopping threads took: 0.0109911 seconds
2018-03-04T09:41:19.524+0000: 21.473: Total time for which application threads were stopped: 0.0001140 seconds, Stopping threads took: 0.0000380 seconds
2018-03-04T09:41:19.733+0000: 21.682: [GC (Allocation Failure) 2018-03-04T09:41:19.734+0000: 21.682: [ParNew
Desired survivor size 4358144 bytes, new threshold 2 (max 6)
- age   1:    3703864 bytes,    3703864 total
- age   2:    3687192 bytes,    7391056 total
: 73842K->8268K(76672K), 0.1340281 secs] 88319K->22745K(1040064K), 0.1341306 secs] [Times: user=0.13 sys=0.01, real=0.14 secs]
2018-03-04T09:41:19.868+0000: 21.816: Total time for which application threads were stopped: 0.1343081 seconds, Stopping threads took: 0.0000328 seconds
2018-03-04T09:41:20.882+0000: 22.831: Total time for which application threads were stopped: 0.0033354 seconds, Stopping threads took: 0.0032585 seconds
2018-03-04T09:41:21.615+0000: 23.563: Total time for which application threads were stopped: 0.0001017 seconds, Stopping threads took: 0.0000310 seconds
2018-03-04T09:41:21.891+0000: 23.840: Total time for which application threads were stopped: 0.0001102 seconds, Stopping threads took: 0.0000349 seconds
2018-03-04T09:41:21.922+0000: 23.870: Total time for which application threads were stopped: 0.0001992 seconds, Stopping threads took: 0.0000606 seconds
2018-03-04T09:41:22.929+0000: 24.878: Total time for which application threads were stopped: 0.0002956 seconds, Stopping threads took: 0.0002015 seconds
2018-03-04T09:41:23.519+0000: 25.468: Total time for which application threads were stopped: 0.0004500 seconds, Stopping threads took: 0.0000288 seconds
2018-03-04T09:41:23.548+0000: 25.497: Total time for which application threads were stopped: 0.0002081 seconds, Stopping threads took: 0.0000123 seconds
2018-03-04T09:41:23.658+0000: 25.607: Total time for which application threads were stopped: 0.0004206 seconds, Stopping threads took: 0.0000295 seconds
2018-03-04T09:41:24.369+0000: 26.318: [GC (Allocation Failure) 2018-03-04T09:41:24.369+0000: 26.318: [ParNew
Desired survivor size 4358144 bytes, new threshold 2 (max 6)
- age   1:    3922416 bytes,    3922416 total
- age   2:    3682208 bytes,    7604624 total
: 76428K->8512K(76672K), 0.1293237 secs] 90905K->33136K(1040064K), 0.1294254 secs] [Times: user=0.13 sys=0.00, real=0.12 secs]
2018-03-04T09:41:24.498+0000: 26.447: Total time for which application threads were stopped: 0.1298352 seconds, Stopping threads took: 0.0000353 seconds
2018-03-04T09:41:24.509+0000: 26.457: Total time for which application threads were stopped: 0.0005516 seconds, Stopping threads took: 0.0004826 seconds
2018-03-04T09:41:24.509+0000: 26.458: Total time for which application threads were stopped: 0.0000598 seconds, Stopping threads took: 0.0000107 seconds
2018-03-04T09:41:25.511+0000: 27.460: Total time for which application threads were stopped: 0.0001928 seconds, Stopping threads took: 0.0001294 seconds
2018-03-04T09:41:26.517+0000: 28.466: Total time for which application threads were stopped: 0.0054144 seconds, Stopping threads took: 0.0053459 seconds
2018-03-04T09:41:27.404+0000: 29.353: Total time for which application threads were stopped: 0.0002320 seconds, Stopping threads took: 0.0000219 seconds
2018-03-04T09:41:27.541+0000: 29.489: Total time for which application threads were stopped: 0.0002008 seconds, Stopping threads took: 0.0000358 seconds
2018-03-04T09:41:27.613+0000: 29.562: Total time for which application threads were stopped: 0.0002273 seconds, Stopping threads took: 0.0000320 seconds
2018-03-04T09:41:27.770+0000: 29.719: Total time for which application threads were stopped: 0.0002513 seconds, Stopping threads took: 0.0000354 seconds
2018-03-04T09:41:27.963+0000: 29.911: Total time for which application threads were stopped: 0.0002453 seconds, Stopping threads took: 0.0000337 seconds
2018-03-04T09:41:28.059+0000: 30.008: Total time for which application threads were stopped: 0.0001522 seconds, Stopping threads took: 0.0000308 seconds
2018-03-04T09:41:28.333+0000: 30.281: Total time for which application threads were stopped: 0.0002204 seconds, Stopping threads took: 0.0000358 seconds
2018-03-04T09:41:28.509+0000: 30.458: [GC (CMS Initial Mark) [1 CMS-initial-mark: 24624K(963392K)] 99184K(1040064K), 0.0832067 secs] [Times: user=0.08 sys=0.00, real=0.08 secs]
2018-03-04T09:41:28.592+0000: 30.541: Total time for which application threads were stopped: 0.0834309 seconds, Stopping threads took: 0.0000432 seconds
2018-03-04T09:41:28.596+0000: 30.545: [CMS-concurrent-mark-start]
2018-03-04T09:41:28.603+0000: 30.551: Total time for which application threads were stopped: 0.0002138 seconds, Stopping threads took: 0.0000156 seconds
2018-03-04T09:41:28.839+0000: 30.788: [CMS-concurrent-mark: 0.243/0.243 secs] [Times: user=0.23 sys=0.01, real=0.25 secs]
2018-03-04T09:41:28.839+0000: 30.788: [GC (Allocation Failure) 2018-03-04T09:41:28.839+0000: 30.788: [ParNew
Desired survivor size 4358144 bytes, new threshold 2 (max 6)
- age   1:    3794000 bytes,    3794000 total
- age   2:    3797232 bytes,    7591232 total
: 76672K->8512K(76672K), 0.1298669 secs] 101296K->44005K(1040064K), 0.1299501 secs] [Times: user=0.13 sys=0.00, real=0.13 secs]
2018-03-04T09:41:28.969+0000: 30.918: Total time for which application threads were stopped: 0.1302093 seconds, Stopping threads took: 0.0000243 seconds
2018-03-04T09:41:28.975+0000: 30.924: [CMS-concurrent-preclean-start]
2018-03-04T09:41:28.998+0000: 30.947: Total time for which application threads were stopped: 0.0001391 seconds, Stopping threads took: 0.0000445 seconds
2018-03-04T09:41:29.079+0000: 31.027: [CMS-concurrent-preclean: 0.098/0.104 secs] [Times: user=0.09 sys=0.02, real=0.11 secs]
2018-03-04T09:41:29.079+0000: 31.027: [CMS-concurrent-abortable-preclean-start]
2018-03-04T09:41:29.211+0000: 31.160: Total time for which application threads were stopped: 0.0002638 seconds, Stopping threads took: 0.0000230 seconds
2018-03-04T09:41:29.410+0000: 31.358: Total time for which application threads were stopped: 0.0004730 seconds, Stopping threads took: 0.0000357 seconds
2018-03-04T09:41:29.614+0000: 31.562: Total time for which application threads were stopped: 0.0030955 seconds, Stopping threads took: 0.0000745 seconds
2018-03-04T09:41:29.878+0000: 31.826: Total time for which application threads were stopped: 0.0001892 seconds, Stopping threads took: 0.0000245 seconds
2018-03-04T09:41:30.215+0000: 32.163: Total time for which application threads were stopped: 0.0001979 seconds, Stopping threads took: 0.0000229 seconds
2018-03-04T09:41:30.455+0000: 32.403: Total time for which application threads were stopped: 0.0001897 seconds, Stopping threads took: 0.0000189 seconds
2018-03-04T09:41:30.686+0000: 32.635: Total time for which application threads were stopped: 0.0002271 seconds, Stopping threads took: 0.0000294 seconds
2018-03-04T09:41:30.873+0000: 32.822: Total time for which application threads were stopped: 0.0002089 seconds, Stopping threads took: 0.0000201 seconds
2018-03-04T09:41:31.065+0000: 33.013: Total time for which application threads were stopped: 0.0001687 seconds, Stopping threads took: 0.0000194 seconds
2018-03-04T09:41:31.266+0000: 33.214: Total time for which application threads were stopped: 0.0001222 seconds, Stopping threads took: 0.0000208 seconds
2018-03-04T09:41:31.433+0000: 33.382: Total time for which application threads were stopped: 0.0002039 seconds, Stopping threads took: 0.0000309 seconds
2018-03-04T09:41:31.462+0000: 33.411: Total time for which application threads were stopped: 0.0148315 seconds, Stopping threads took: 0.0146897 seconds
2018-03-04T09:41:31.473+0000: 33.422: Total time for which application threads were stopped: 0.0000946 seconds, Stopping threads took: 0.0000136 seconds
2018-03-04T09:41:31.578+0000: 33.527: Total time for which application threads were stopped: 0.0002107 seconds, Stopping threads took: 0.0000209 seconds
2018-03-04T09:41:31.734+0000: 33.683: Total time for which application threads were stopped: 0.0002321 seconds, Stopping threads took: 0.0000225 seconds
2018-03-04T09:41:31.774+0000: 33.723: Total time for which application threads were stopped: 0.0002281 seconds, Stopping threads took: 0.0000320 seconds
2018-03-04T09:41:31.774+0000: 33.723: [CMS-concurrent-abortable-preclean: 2.497/2.696 secs] [Times: user=2.59 sys=0.08, real=2.69 secs]
2018-03-04T09:41:31.775+0000: 33.723: [GC (CMS Final Remark) [YG occupancy: 42807 K (76672 K)]2018-03-04T09:41:31.775+0000: 33.723: [Rescan (parallel) , 0.1021795 secs]2018-03-04T09:41:31.877+0000: 33.826: [weak refs processing, 0.0003390 secs]2018-03-04T09:41:31.877+0000: 33.826: [class unloading, 0.0111743 secs]2018-03-04T09:41:31.889+0000: 33.837: [scrub symbol table, 0.0168819 secs]2018-03-04T09:41:31.906+0000: 33.854: [scrub string table, 0.0009940 secs][1 CMS-remark: 35493K(963392K)] 78301K(1040064K), 0.1320860 secs] [Times: user=0.13 sys=0.00, real=0.14 secs]
2018-03-04T09:41:31.907+0000: 33.856: Total time for which application threads were stopped: 0.1324285 seconds, Stopping threads took: 0.0002622 seconds
2018-03-04T09:41:31.922+0000: 33.870: [CMS-concurrent-sweep-start]
2018-03-04T09:41:31.939+0000: 33.887: Total time for which application threads were stopped: 0.0006150 seconds, Stopping threads took: 0.0004978 seconds
2018-03-04T09:41:31.993+0000: 33.942: [CMS-concurrent-sweep: 0.070/0.071 secs] [Times: user=0.04 sys=0.00, real=0.07 secs]
2018-03-04T09:41:31.993+0000: 33.942: [CMS-concurrent-reset-start]
2018-03-04T09:41:32.001+0000: 33.950: [CMS-concurrent-reset: 0.008/0.008 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2018-03-04T09:41:32.002+0000: 33.950: Total time for which application threads were stopped: 0.0005549 seconds, Stopping threads took: 0.0000370 seconds
2018-03-04T09:41:32.056+0000: 34.004: Total time for which application threads were stopped: 0.0001034 seconds, Stopping threads took: 0.0000294 seconds
2018-03-04T09:41:32.056+0000: 34.005: Total time for which application threads were stopped: 0.0000646 seconds, Stopping threads took: 0.0000206 seconds
2018-03-04T09:41:32.057+0000: 34.006: Total time for which application threads were stopped: 0.0000651 seconds, Stopping threads took: 0.0000210 seconds
2018-03-04T09:41:32.057+0000: 34.006: Total time for which application threads were stopped: 0.0000687 seconds, Stopping threads took: 0.0000224 seconds
2018-03-04T09:41:32.058+0000: 34.006: Total time for which application threads were stopped: 0.0000681 seconds, Stopping threads took: 0.0000200 seconds
2018-03-04T09:41:32.058+0000: 34.006: Total time for which application threads were stopped: 0.0000597 seconds, Stopping threads took: 0.0000169 seconds
2018-03-04T09:41:32.058+0000: 34.007: Total time for which application threads were stopped: 0.0001896 seconds, Stopping threads took: 0.0001428 seconds
2018-03-04T09:41:32.058+0000: 34.007: Total time for which application threads were stopped: 0.0000588 seconds, Stopping threads took: 0.0000172 seconds
2018-03-04T09:41:32.058+0000: 34.007: Total time for which application threads were stopped: 0.0000580 seconds, Stopping threads took: 0.0000169 seconds
2018-03-04T09:41:32.058+0000: 34.007: Total time for which application threads were stopped: 0.0000629 seconds, Stopping threads took: 0.0000177 seconds
2018-03-04T09:41:32.059+0000: 34.007: Total time for which application threads were stopped: 0.0000656 seconds, Stopping threads took: 0.0000171 seconds
2018-03-04T09:41:32.059+0000: 34.007: Total time for which application threads were stopped: 0.0000622 seconds, Stopping threads took: 0.0000175 seconds
2018-03-04T09:41:32.059+0000: 34.007: Total time for which application threads were stopped: 0.0000593 seconds, Stopping threads took: 0.0000164 seconds
2018-03-04T09:41:32.059+0000: 34.008: Total time for which application threads were stopped: 0.0000625 seconds, Stopping threads took: 0.0000172 seconds
2018-03-04T09:41:32.064+0000: 34.012: Total time for which application threads were stopped: 0.0000968 seconds, Stopping threads took: 0.0000309 seconds
2018-03-04T09:41:32.064+0000: 34.013: Total time for which application threads were stopped: 0.0000538 seconds, Stopping threads took: 0.0000102 seconds
2018-03-04T09:41:32.064+0000: 34.013: Total time for which application threads were stopped: 0.0000524 seconds, Stopping threads took: 0.0000097 seconds
2018-03-04T09:41:32.065+0000: 34.013: Total time for which application threads were stopped: 0.0000578 seconds, Stopping threads took: 0.0000098 seconds
2018-03-04T09:41:32.065+0000: 34.013: Total time for which application threads were stopped: 0.0000502 seconds, Stopping threads took: 0.0000077 seconds
2018-03-04T09:41:32.065+0000: 34.013: Total time for which application threads were stopped: 0.0000603 seconds, Stopping threads took: 0.0000165 seconds
2018-03-04T09:41:32.065+0000: 34.014: Total time for which application threads were stopped: 0.0000550 seconds, Stopping threads took: 0.0000103 seconds
2018-03-04T09:41:32.066+0000: 34.014: Total time for which application threads were stopped: 0.0000531 seconds, Stopping threads took: 0.0000105 seconds
2018-03-04T09:41:32.066+0000: 34.014: Total time for which application threads were stopped: 0.0000518 seconds, Stopping threads took: 0.0000093 seconds
2018-03-04T09:41:32.066+0000: 34.015: Total time for which application threads were stopped: 0.0000463 seconds, Stopping threads took: 0.0000078 seconds
2018-03-04T09:41:32.067+0000: 34.015: Total time for which application threads were stopped: 0.0000657 seconds, Stopping threads took: 0.0000209 seconds
2018-03-04T09:41:32.067+0000: 34.016: Total time for which application threads were stopped: 0.0000835 seconds, Stopping threads took: 0.0000218 seconds
2018-03-04T09:41:32.068+0000: 34.016: Total time for which application threads were stopped: 0.0000591 seconds, Stopping threads took: 0.0000175 seconds
2018-03-04T09:41:32.068+0000: 34.016: Total time for which application threads were stopped: 0.0000593 seconds, Stopping threads took: 0.0000175 seconds
2018-03-04T09:41:32.068+0000: 34.016: Total time for which application threads were stopped: 0.0000570 seconds, Stopping threads took: 0.0000168 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000594 seconds, Stopping threads took: 0.0000171 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000569 seconds, Stopping threads took: 0.0000166 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000584 seconds, Stopping threads took: 0.0000167 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000569 seconds, Stopping threads took: 0.0000164 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000593 seconds, Stopping threads took: 0.0000172 seconds
2018-03-04T09:41:32.068+0000: 34.017: Total time for which application threads were stopped: 0.0000592 seconds, Stopping threads took: 0.0000173 seconds
2018-03-04T09:41:32.069+0000: 34.017: Total time for which application threads were stopped: 0.0000573 seconds, Stopping threads took: 0.0000169 seconds
2018-03-04T09:41:32.069+0000: 34.017: Total time for which application threads were stopped: 0.0000598 seconds, Stopping threads took: 0.0000173 seconds
2018-03-04T09:41:32.069+0000: 34.017: Total time for which application threads were stopped: 0.0000572 seconds, Stopping threads took: 0.0000165 seconds
2018-03-04T09:41:32.069+0000: 34.018: Total time for which application threads were stopped: 0.0000476 seconds, Stopping threads took: 0.0000076 seconds
2018-03-04T09:41:32.069+0000: 34.018: Total time for which application threads were stopped: 0.0000491 seconds, Stopping threads took: 0.0000078 seconds
2018-03-04T09:41:32.069+0000: 34.018: Total time for which application threads were stopped: 0.0000471 seconds, Stopping threads took: 0.0000077 seconds
2018-03-04T09:41:32.084+0000: 34.032: Total time for which application threads were stopped: 0.0000942 seconds, Stopping threads took: 0.0000302 seconds
2018-03-04T09:41:32.110+0000: 34.058: Total time for which application threads were stopped: 0.0024743 seconds, Stopping threads took: 0.0024005 seconds
2018-03-04T09:41:32.116+0000: 34.065: Total time for which application threads were stopped: 0.0001111 seconds, Stopping threads took: 0.0000532 seconds
2018-03-04T09:41:32.117+0000: 34.065: Total time for which application threads were stopped: 0.0000775 seconds, Stopping threads took: 0.0000188 seconds
2018-03-04T09:41:32.136+0000: 34.084: Total time for which application threads were stopped: 0.0004719 seconds, Stopping threads took: 0.0000285 seconds
2018-03-04T09:41:32.311+0000: 34.260: Total time for which application threads were stopped: 0.0005102 seconds, Stopping threads took: 0.0003735 seconds
2018-03-04T09:41:32.314+0000: 34.263: Total time for which application threads were stopped: 0.0000602 seconds, Stopping threads took: 0.0000130 seconds
2018-03-04T09:41:32.314+0000: 34.263: Total time for which application threads were stopped: 0.0000495 seconds, Stopping threads took: 0.0000110 seconds
2018-03-04T09:41:32.315+0000: 34.264: Total time for which application threads were stopped: 0.0000562 seconds, Stopping threads took: 0.0000124 seconds
2018-03-04T09:41:32.330+0000: 34.278: Total time for which application threads were stopped: 0.0001402 seconds, Stopping threads took: 0.0000239 seconds
2018-03-04T09:41:32.330+0000: 34.279: Total time for which application threads were stopped: 0.0000786 seconds, Stopping threads took: 0.0000282 seconds
2018-03-04T09:41:32.330+0000: 34.279: Total time for which application threads were stopped: 0.0000598 seconds, Stopping threads took: 0.0000185 seconds
2018-03-04T09:41:32.333+0000: 34.281: Total time for which application threads were stopped: 0.0000816 seconds, Stopping threads took: 0.0000228 seconds
2018-03-04T09:41:32.334+0000: 34.282: Total time for which application threads were stopped: 0.0000691 seconds, Stopping threads took: 0.0000228 seconds
2018-03-04T09:41:32.334+0000: 34.283: Total time for which application threads were stopped: 0.0000581 seconds, Stopping threads took: 0.0000190 seconds
2018-03-04T09:41:32.334+0000: 34.283: Total time for which application threads were stopped: 0.0000634 seconds, Stopping threads took: 0.0000228 seconds
2018-03-04T09:41:32.369+0000: 34.318: Total time for which application threads were stopped: 0.0097693 seconds, Stopping threads took: 0.0097055 seconds
2018-03-04T09:41:32.375+0000: 34.323: Total time for which application threads were stopped: 0.0000775 seconds, Stopping threads took: 0.0000251 seconds
2018-03-04T09:41:32.375+0000: 34.324: Total time for which application threads were stopped: 0.0000576 seconds, Stopping threads took: 0.0000191 seconds
2018-03-04T09:41:32.384+0000: 34.333: Total time for which application threads were stopped: 0.0000647 seconds, Stopping threads took: 0.0000136 seconds
2018-03-04T09:41:32.399+0000: 34.347: Total time for which application threads were stopped: 0.0039581 seconds, Stopping threads took: 0.0000248 seconds
2018-03-04T09:41:32.410+0000: 34.359: Total time for which application threads were stopped: 0.0020943 seconds, Stopping threads took: 0.0020346 seconds
2018-03-04T09:41:32.413+0000: 34.362: Total time for which application threads were stopped: 0.0000662 seconds, Stopping threads took: 0.0000139 seconds
2018-03-04T09:41:32.413+0000: 34.362: Total time for which application threads were stopped: 0.0000761 seconds, Stopping threads took: 0.0000120 seconds
2018-03-04T09:41:32.419+0000: 34.368: Total time for which application threads were stopped: 0.0006705 seconds, Stopping threads took: 0.0006141 seconds
2018-03-04T09:41:32.428+0000: 34.377: Total time for which application threads were stopped: 0.0001864 seconds, Stopping threads took: 0.0000272 seconds
2018-03-04T09:41:32.430+0000: 34.379: Total time for which application threads were stopped: 0.0001418 seconds, Stopping threads took: 0.0000299 seconds
2018-03-04T09:41:32.431+0000: 34.380: Total time for which application threads were stopped: 0.0001271 seconds, Stopping threads took: 0.0000230 seconds
2018-03-04T09:41:32.439+0000: 34.388: Total time for which application threads were stopped: 0.0001660 seconds, Stopping threads took: 0.0001006 seconds
2018-03-04T09:41:32.440+0000: 34.388: Total time for which application threads were stopped: 0.0001026 seconds, Stopping threads took: 0.0000121 seconds
2018-03-04T09:41:32.440+0000: 34.389: Total time for which application threads were stopped: 0.0000745 seconds, Stopping threads took: 0.0000114 seconds
2018-03-04T09:41:32.440+0000: 34.389: Total time for which application threads were stopped: 0.0000703 seconds, Stopping threads took: 0.0000101 seconds
2018-03-04T09:41:32.453+0000: 34.402: Total time for which application threads were stopped: 0.0000993 seconds, Stopping threads took: 0.0000234 seconds
2018-03-04T09:41:32.453+0000: 34.402: Total time for which application threads were stopped: 0.0000776 seconds, Stopping threads took: 0.0000250 seconds
2018-03-04T09:41:32.454+0000: 34.402: Total time for which application threads were stopped: 0.0000778 seconds, Stopping threads took: 0.0000267 seconds
2018-03-04T09:41:32.458+0000: 34.406: Total time for which application threads were stopped: 0.0000903 seconds, Stopping threads took: 0.0000365 seconds
2018-03-04T09:41:32.458+0000: 34.407: Total time for which application threads were stopped: 0.0000755 seconds, Stopping threads took: 0.0000116 seconds
2018-03-04T09:41:32.458+0000: 34.407: Total time for which application threads were stopped: 0.0000875 seconds, Stopping threads took: 0.0000104 seconds
2018-03-04T09:41:32.458+0000: 34.407: Total time for which application threads were stopped: 0.0000971 seconds, Stopping threads took: 0.0000091 seconds
2018-03-04T09:41:32.459+0000: 34.407: Total time for which application threads were stopped: 0.0000613 seconds, Stopping threads took: 0.0000093 seconds
2018-03-04T09:41:32.459+0000: 34.407: Total time for which application threads were stopped: 0.0000690 seconds, Stopping threads took: 0.0000099 seconds
2018-03-04T09:41:32.459+0000: 34.407: Total time for which application threads were stopped: 0.0000616 seconds, Stopping threads took: 0.0000086 seconds
2018-03-04T09:41:32.459+0000: 34.408: Total time for which application threads were stopped: 0.0001872 seconds, Stopping threads took: 0.0001169 seconds
2018-03-04T09:41:32.460+0000: 34.408: Total time for which application threads were stopped: 0.0000790 seconds, Stopping threads took: 0.0000126 seconds
2018-03-04T09:41:32.460+0000: 34.409: Total time for which application threads were stopped: 0.0000768 seconds, Stopping threads took: 0.0000123 seconds
2018-03-04T09:41:32.463+0000: 34.412: Total time for which application threads were stopped: 0.0020491 seconds, Stopping threads took: 0.0019817 seconds
2018-03-04T09:41:32.463+0000: 34.412: Total time for which application threads were stopped: 0.0000749 seconds, Stopping threads took: 0.0000111 seconds
2018-03-04T09:41:32.463+0000: 34.412: Total time for which application threads were stopped: 0.0000661 seconds, Stopping threads took: 0.0000088 seconds
2018-03-04T09:41:32.464+0000: 34.412: Total time for which application threads were stopped: 0.0000668 seconds, Stopping threads took: 0.0000087 seconds
2018-03-04T09:41:32.492+0000: 34.440: Total time for which application threads were stopped: 0.0001399 seconds, Stopping threads took: 0.0000261 seconds
2018-03-04T09:41:32.492+0000: 34.441: Total time for which application threads were stopped: 0.0001125 seconds, Stopping threads took: 0.0000194 seconds
2018-03-04T09:41:32.493+0000: 34.442: Total time for which application threads were stopped: 0.0000914 seconds, Stopping threads took: 0.0000262 seconds
2018-03-04T09:41:32.493+0000: 34.442: Total time for which application threads were stopped: 0.0000831 seconds, Stopping threads took: 0.0000219 seconds
2018-03-04T09:41:32.521+0000: 34.470: Total time for which application threads were stopped: 0.0001019 seconds, Stopping threads took: 0.0000309 seconds
2018-03-04T09:41:32.521+0000: 34.470: Total time for which application threads were stopped: 0.0001214 seconds, Stopping threads took: 0.0000232 seconds
2018-03-04T09:41:32.522+0000: 34.471: Total time for which application threads were stopped: 0.0000907 seconds, Stopping threads took: 0.0000228 seconds
2018-03-04T09:41:32.541+0000: 34.490: Total time for which application threads were stopped: 0.0001072 seconds, Stopping threads took: 0.0000258 seconds
2018-03-04T09:41:32.548+0000: 34.497: Total time for which application threads were stopped: 0.0000848 seconds, Stopping threads took: 0.0000246 seconds
2018-03-04T09:41:32.549+0000: 34.498: Total time for which application threads were stopped: 0.0000842 seconds, Stopping threads took: 0.0000129 seconds
2018-03-04T09:41:32.549+0000: 34.498: Total time for which application threads were stopped: 0.0000715 seconds, Stopping threads took: 0.0000122 seconds
2018-03-04T09:41:32.550+0000: 34.498: Total time for which application threads were stopped: 0.0000617 seconds, Stopping threads took: 0.0000138 seconds
2018-03-04T09:41:32.550+0000: 34.498: Total time for which application threads were stopped: 0.0000573 seconds, Stopping threads took: 0.0000101 seconds
2018-03-04T09:41:32.550+0000: 34.498: Total time for which application threads were stopped: 0.0000601 seconds, Stopping threads took: 0.0000111 seconds
2018-03-04T09:41:32.550+0000: 34.499: Total time for which application threads were stopped: 0.0000589 seconds, Stopping threads took: 0.0000104 seconds
2018-03-04T09:41:32.550+0000: 34.499: Total time for which application threads were stopped: 0.0000715 seconds, Stopping threads took: 0.0000124 seconds
2018-03-04T09:41:32.553+0000: 34.501: Total time for which application threads were stopped: 0.0000999 seconds, Stopping threads took: 0.0000305 seconds
2018-03-04T09:41:32.553+0000: 34.502: Total time for which application threads were stopped: 0.0000682 seconds, Stopping threads took: 0.0000196 seconds
2018-03-04T09:41:32.553+0000: 34.502: Total time for which application threads were stopped: 0.0000698 seconds, Stopping threads took: 0.0000219 seconds
2018-03-04T09:41:32.553+0000: 34.502: Total time for which application threads were stopped: 0.0000704 seconds, Stopping threads took: 0.0000207 seconds
2018-03-04T09:41:32.553+0000: 34.502: Total time for which application threads were stopped: 0.0000679 seconds, Stopping threads took: 0.0000197 seconds
2018-03-04T09:41:32.554+0000: 34.502: Total time for which application threads were stopped: 0.0000737 seconds, Stopping threads took: 0.0000218 seconds
2018-03-04T09:41:32.559+0000: 34.507: Total time for which application threads were stopped: 0.0000877 seconds, Stopping threads took: 0.0000320 seconds
2018-03-04T09:41:32.561+0000: 34.510: Total time for which application threads were stopped: 0.0000858 seconds, Stopping threads took: 0.0000244 seconds
2018-03-04T09:41:32.563+0000: 34.511: Total time for which application threads were stopped: 0.0000760 seconds, Stopping threads took: 0.0000235 seconds
2018-03-04T09:41:32.564+0000: 34.512: Total time for which application threads were stopped: 0.0000770 seconds, Stopping threads took: 0.0000242 seconds
2018-03-04T09:41:32.565+0000: 34.513: Total time for which application threads were stopped: 0.0001141 seconds, Stopping threads took: 0.0000234 seconds
2018-03-04T09:41:32.565+0000: 34.514: Total time for which application threads were stopped: 0.0000702 seconds, Stopping threads took: 0.0000227 seconds
2018-03-04T09:41:33.377+0000: 35.326: Total time for which application threads were stopped: 0.0002310 seconds, Stopping threads took: 0.0000374 seconds
2018-03-04T09:41:33.397+0000: 35.345: Total time for which application threads were stopped: 0.0117636 seconds, Stopping threads took: 0.0116375 seconds
2018-03-04T09:41:33.399+0000: 35.348: Total time for which application threads were stopped: 0.0000836 seconds, Stopping threads took: 0.0000176 seconds
2018-03-04T09:41:33.400+0000: 35.348: Total time for which application threads were stopped: 0.0000708 seconds, Stopping threads took: 0.0000123 seconds
2018-03-04T09:41:33.400+0000: 35.349: Total time for which application threads were stopped: 0.0000688 seconds, Stopping threads took: 0.0000115 seconds
2018-03-04T09:41:33.400+0000: 35.349: Total time for which application threads were stopped: 0.0000762 seconds, Stopping threads took: 0.0000186 seconds
2018-03-04T09:41:33.400+0000: 35.349: Total time for which application threads were stopped: 0.0001846 seconds, Stopping threads took: 0.0000103 seconds
2018-03-04T09:41:33.401+0000: 35.349: Total time for which application threads were stopped: 0.0000700 seconds, Stopping threads took: 0.0000114 seconds
2018-03-04T09:41:33.402+0000: 35.350: Total time for which application threads were stopped: 0.0000730 seconds, Stopping threads took: 0.0000127 seconds
2018-03-04T09:41:33.407+0000: 35.355: Total time for which application threads were stopped: 0.0000968 seconds, Stopping threads took: 0.0000292 seconds
2018-03-04T09:41:33.407+0000: 35.356: Total time for which application threads were stopped: 0.0000737 seconds, Stopping threads took: 0.0000129 seconds
2018-03-04T09:41:33.408+0000: 35.357: Total time for which application threads were stopped: 0.0000691 seconds, Stopping threads took: 0.0000127 seconds
2018-03-04T09:41:33.409+0000: 35.358: Total time for which application threads were stopped: 0.0000612 seconds, Stopping threads took: 0.0000129 seconds
2018-03-04T09:41:34.355+0000: 36.304: Total time for which application threads were stopped: 0.0001302 seconds, Stopping threads took: 0.0000326 seconds
2018-03-04T09:41:34.356+0000: 36.305: Total time for which application threads were stopped: 0.0000795 seconds, Stopping threads took: 0.0000244 seconds
2018-03-04T09:41:34.356+0000: 36.305: Total time for which application threads were stopped: 0.0000610 seconds, Stopping threads took: 0.0000114 seconds
2018-03-04T09:41:34.357+0000: 36.305: Total time for which application threads were stopped: 0.0000634 seconds, Stopping threads took: 0.0000122 seconds
2018-03-04T09:41:34.358+0000: 36.306: Total time for which application threads were stopped: 0.0000670 seconds, Stopping threads took: 0.0000130 seconds
2018-03-04T09:41:34.358+0000: 36.307: Total time for which application threads were stopped: 0.0001130 seconds, Stopping threads took: 0.0000278 seconds
2018-03-04T09:41:34.359+0000: 36.307: Total time for which application threads were stopped: 0.0000612 seconds, Stopping threads took: 0.0000119 seconds
2018-03-04T09:41:34.359+0000: 36.308: Total time for which application threads were stopped: 0.0000703 seconds, Stopping threads took: 0.0000136 seconds
2018-03-04T09:41:34.360+0000: 36.308: Total time for which application threads were stopped: 0.0000662 seconds, Stopping threads took: 0.0000127 seconds
2018-03-04T09:41:34.361+0000: 36.310: Total time for which application threads were stopped: 0.0000769 seconds, Stopping threads took: 0.0000245 seconds
2018-03-04T09:41:34.362+0000: 36.310: Total time for which application threads were stopped: 0.0000608 seconds, Stopping threads took: 0.0000116 seconds
2018-03-04T09:41:34.362+0000: 36.310: Total time for which application threads were stopped: 0.0000634 seconds, Stopping threads took: 0.0000125 seconds
2018-03-04T09:41:34.362+0000: 36.311: Total time for which application threads were stopped: 0.0000619 seconds, Stopping threads took: 0.0000127 seconds
2018-03-04T09:41:34.363+0000: 36.312: Total time for which application threads were stopped: 0.0000751 seconds, Stopping threads took: 0.0000237 seconds
2018-03-04T09:41:34.363+0000: 36.312: Total time for which application threads were stopped: 0.0000603 seconds, Stopping threads took: 0.0000114 seconds
2018-03-04T09:41:34.364+0000: 36.312: Total time for which application threads were stopped: 0.0000637 seconds, Stopping threads took: 0.0000125 seconds
2018-03-04T09:41:34.364+0000: 36.313: Total time for which application threads were stopped: 0.0000891 seconds, Stopping threads took: 0.0000120 seconds
2018-03-04T09:41:35.364+0000: 37.312: Total time for which application threads were stopped: 0.0001238 seconds, Stopping threads took: 0.0000390 seconds
2018-03-04T09:41:35.389+0000: 37.338: Total time for which application threads were stopped: 0.0000845 seconds, Stopping threads took: 0.0000246 seconds
2018-03-04T09:41:35.389+0000: 37.338: Total time for which application threads were stopped: 0.0000682 seconds, Stopping threads took: 0.0000211 seconds
2018-03-04T09:41:35.552+0000: 37.500: Total time for which application threads were stopped: 0.0001356 seconds, Stopping threads took: 0.0000370 seconds
2018-03-04T09:41:35.553+0000: 37.502: Total time for which application threads were stopped: 0.0000865 seconds, Stopping threads took: 0.0000288 seconds
2018-03-04T09:41:35.571+0000: 37.519: Total time for which application threads were stopped: 0.0054710 seconds, Stopping threads took: 0.0051047 seconds
2018-03-04T09:41:35.575+0000: 37.524: Total time for which application threads were stopped: 0.0000928 seconds, Stopping threads took: 0.0000145 seconds
2018-03-04T09:41:35.591+0000: 37.540: Total time for which application threads were stopped: 0.0001335 seconds, Stopping threads took: 0.0000267 seconds
2018-03-04T09:41:35.591+0000: 37.540: Total time for which application threads were stopped: 0.0001087 seconds, Stopping threads took: 0.0000200 seconds
2018-03-04T09:41:35.591+0000: 37.540: Total time for which application threads were stopped: 0.0001135 seconds, Stopping threads took: 0.0000220 seconds
2018-03-04T09:41:35.592+0000: 37.540: Total time for which application threads were stopped: 0.0001013 seconds, Stopping threads took: 0.0000181 seconds
2018-03-04T09:41:35.625+0000: 37.574: Total time for which application threads were stopped: 0.0022043 seconds, Stopping threads took: 0.0021391 seconds
2018-03-04T09:41:35.634+0000: 37.583: Total time for which application threads were stopped: 0.0044460 seconds, Stopping threads took: 0.0043790 seconds
2018-03-04T09:41:35.635+0000: 37.583: Total time for which application threads were stopped: 0.0000704 seconds, Stopping threads took: 0.0000126 seconds
2018-03-04T09:41:35.642+0000: 37.591: Total time for which application threads were stopped: 0.0001286 seconds, Stopping threads took: 0.0000447 seconds
2018-03-04T09:41:36.642+0000: 38.591: Total time for which application threads were stopped: 0.0001323 seconds, Stopping threads took: 0.0000317 seconds
2018-03-04T09:41:40.933+0000: 42.882: Total time for which application threads were stopped: 0.0001146 seconds, Stopping threads took: 0.0000309 seconds
2018-03-04T09:41:44.934+0000: 46.882: Total time for which application threads were stopped: 0.0001139 seconds, Stopping threads took: 0.0000335 seconds
2018-03-04T09:41:45.934+0000: 47.882: Total time for which application threads were stopped: 0.0001117 seconds, Stopping threads took: 0.0000333 seconds
2018-03-04T09:41:46.934+0000: 48.883: Total time for which application threads were stopped: 0.0001307 seconds, Stopping threads took: 0.0000485 seconds
2018-03-04T09:42:05.582+0000: 67.531: Total time for which application threads were stopped: 0.0001189 seconds, Stopping threads took: 0.0000351 seconds
2018-03-04T09:42:06.582+0000: 68.531: Total time for which application threads were stopped: 0.0001183 seconds, Stopping threads took: 0.0000403 seconds
2018-03-04T09:42:10.583+0000: 72.532: Total time for which application threads were stopped: 0.0001236 seconds, Stopping threads took: 0.0000370 seconds
2018-03-04T09:43:04.589+0000: 126.538: Total time for which application threads were stopped: 0.0001380 seconds, Stopping threads took: 0.0000577 seconds
2018-03-04T09:43:21.592+0000: 143.540: Total time for which application threads were stopped: 0.0001169 seconds, Stopping threads took: 0.0000350 seconds
2018-03-04T09:43:36.594+0000: 158.542: Total time for which application threads were stopped: 0.0001513 seconds, Stopping threads took: 0.0000586 seconds
2018-03-04T09:43:41.594+0000: 163.543: Total time for which application threads were stopped: 0.0001249 seconds, Stopping threads took: 0.0000353 seconds
2018-03-04T09:44:01.597+0000: 183.545: Total time for which application threads were stopped: 0.0001048 seconds, Stopping threads took: 0.0000284 seconds
2018-03-04T09:44:07.598+0000: 189.546: Total time for which application threads were stopped: 0.0001430 seconds, Stopping threads took: 0.0000582 seconds
2018-03-04T09:44:27.600+0000: 209.549: Total time for which application threads were stopped: 0.0001346 seconds, Stopping threads took: 0.0000381 seconds
2018-03-04T09:44:28.600+0000: 210.549: Total time for which application threads were stopped: 0.0001253 seconds, Stopping threads took: 0.0000344 seconds
2018-03-04T09:44:41.602+0000: 223.551: Total time for which application threads were stopped: 0.0001491 seconds, Stopping threads took: 0.0000634 seconds
2018-03-04T09:44:42.602+0000: 224.551: Total time for which application threads were stopped: 0.0001469 seconds, Stopping threads took: 0.0000336 seconds
2018-03-04T09:44:55.604+0000: 237.553: Total time for which application threads were stopped: 0.0001597 seconds, Stopping threads took: 0.0000357 seconds
2018-03-04T09:45:06.606+0000: 248.554: Total time for which application threads were stopped: 0.0001453 seconds, Stopping threads took: 0.0000587 seconds
2018-03-04T09:45:14.608+0000: 256.557: Total time for which application threads were stopped: 0.0001343 seconds, Stopping threads took: 0.0000417 seconds
2018-03-04T09:45:28.610+0000: 270.559: Total time for which application threads were stopped: 0.0001446 seconds, Stopping threads took: 0.0000438 seconds
2018-03-04T09:45:31.611+0000: 273.559: Total time for which application threads were stopped: 0.0001237 seconds, Stopping threads took: 0.0000341 seconds
2018-03-04T09:46:11.615+0000: 313.564: Total time for which application threads were stopped: 0.0001536 seconds, Stopping threads took: 0.0000641 seconds
2018-03-04T09:46:36.619+0000: 338.567: Total time for which application threads were stopped: 0.0001620 seconds, Stopping threads took: 0.0000598 seconds

Is this sufficient?

Thanks again :)

nshou commented 5 years ago

(Original comment by nshou on 2018-03-04T16:00:38.631960+00:00)

As far as I see the logs, both cases should be correctly operating Elasticsearch without errors. You can put, retrieve, and search data in it, but it can be confirmed that Azure VM has some delay to build up the system in the container.

As for Digital Ocean VM, you can find it took only 8 seconds to finish starting up Elasticsearch and no excessive Full GCs (a.k.a. stop-the-world) occurred. The system should work fine as a whole in that VM.

On the other hand, it took 29 seconds to boot up Elasticsearch process in Azure VM, which means that it was not able to provide any services during that term and Kibana process alerted stating that it was unable to connect Elasticsearch via 9200 port. In this case, the delay was mainly caused by its low memory. Although Full GCs became calm after opening the service, they were needed to be extremely called during its preparation phase, making the phase itself slow down.

I believe that installing more memory (or taking the measure mentioned in issue #1) would solve the problems you faced.

nshou commented 5 years ago

(Original comment by nshou on 2018-03-04T16:05:03.730660+00:00)

And I would like to hear from the first anonymous poster about the situation after that, too :)

nshou commented 5 years ago

(Original comment by nathandaly on 2018-03-04T22:00:09.065193+00:00)

Sorry the OP is myself but to comment after that I had to be logged in.

It is probably a resource issue then. Thanks for your help. You can mark it as a resolved :)

nshou commented 5 years ago

(Original comment by nshou on 2018-03-05T14:19:59.867515+00:00)

It's my pleasure.