search

300Geeks / aws-elk-billing

Tools for Loading and Visualising AWS Detailed Billing with ELK(Elasticsearch, Logstash, Kibana)
MIT License
157 stars 26 forks source link

Elasticsearch plugin is red #24

Closed kenkit closed 6 years ago

kenkit commented 6 years ago

I've setup everything well but it seems elastic search just isn't working. I get Unable to connect to Elasticsearch at http://elasticsearch:9200.

droidlabour commented 6 years ago

@kenkit Can you post console output of docker-compose up --build

kenkit commented 6 years ago

It's pretty long

~/Desktop/aws-elk-billing$ sudo docker-compose up --build
Building aws-elk-billing
Step 1/21 : FROM ubuntu:14.04.4
 ---> 0ccb13bf1954
Step 2/21 : RUN apt-get update && apt-get install -y --no-install-recommends   g++              gcc             libc6-dev               make    && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 3f5a0b2a85c0
Step 3/21 : ENV GOLANG_VERSION 1.6.2
 ---> Using cache
 ---> 23301742eb31
Step 4/21 : ENV GOLANG_DOWNLOAD_URL https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz
 ---> Using cache
 ---> bf40d663de25
Step 5/21 : ENV GOLANG_DOWNLOAD_SHA256 e40c36ae71756198478624ed1bb4ce17597b3c19d243f3f0899bb5740d56212a
 ---> Using cache
 ---> 455f76ada064
Step 6/21 : RUN apt-get update
 ---> Using cache
 ---> 4bb970545dc9
Step 7/21 : RUN apt-get install -y curl
 ---> Using cache
 ---> 32e25d713e66
Step 8/21 : RUN curl -fsSL "$GOLANG_DOWNLOAD_URL" -o golang.tar.gz      && echo "$GOLANG_DOWNLOAD_SHA256  golang.tar.gz" | sha256sum -c -       && tar -C /usr/local -xzf golang.tar.gz         && rm golang.tar.gz
 ---> Using cache
 ---> 2b55e063c000
Step 9/21 : ENV GOPATH /go
 ---> Using cache
 ---> adcc8d258d73
Step 10/21 : ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
 ---> Using cache
 ---> e768bee8d4f5
Step 11/21 : RUN mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"
 ---> Using cache
 ---> 7256effd44e2
Step 12/21 : COPY go-wrapper /usr/local/bin/
 ---> Using cache
 ---> 6f89ab31bb23
Step 13/21 : RUN apt-get install -y python
 ---> Using cache
 ---> becd0fa621d3
Step 14/21 : RUN apt-get install -y python-pip
 ---> Using cache
 ---> 0ef12d36d5e9
Step 15/21 : RUN pip install boto3
 ---> Using cache
 ---> 3a49bf1da39f
Step 16/21 : RUN pip install pyelasticsearch
 ---> Using cache
 ---> 43a2bd3d78d8
Step 17/21 : RUN pip install nose
 ---> Using cache
 ---> 7baa2061c26f
Step 18/21 : WORKDIR /aws-elk-billing
 ---> Using cache
 ---> 50a85cf064f2
Step 19/21 : ENV TZ Asia/Kolkata
 ---> Using cache
 ---> 77321e90cf5f
Step 20/21 : RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 ---> Using cache
 ---> eb38a7f77e3e
Step 21/21 : CMD tail -f /dev/null
 ---> Using cache
 ---> ea41a60df090
Successfully built ea41a60df090
Successfully tagged awselkbilling_aws-elk-billing:latest
Starting awselkbilling_elasticsearch_1 ...
Starting awselkbilling_elasticsearch_1 ... done
Starting awselkbilling_kibana_1 ...
Starting awselkbilling_kibana_1
Starting awselkbilling_logstash_1 ...
Starting awselkbilling_logstash_1 ... done
Starting awselkbilling_aws-elk-billing_1 ...
Starting awselkbilling_aws-elk-billing_1 ... done
Attaching to awselkbilling_elasticsearch_1, awselkbilling_kibana_1, awselkbilling_logstash_1, awselkbilling_aws-elk-billing_1
elasticsearch_1    | OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5330000, 986513408, 0) failed; error='Out of memory' (errno=12)
elasticsearch_1    | #
elasticsearch_1    | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch_1    | # Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
elasticsearch_1    | # Can not save log file, dump to screen..
elasticsearch_1    | #
elasticsearch_1    | # There is insufficient memory for the Java Runtime Environment to continue.
elasticsearch_1    | # Native memory allocation (mmap) failed to map 986513408 bytes for committing reserved memory.
elasticsearch_1    | # Possible reasons:
elasticsearch_1    | #   The system is out of physical RAM or swap space
elasticsearch_1    | #   In 32 bit mode, the process size limit was hit
elasticsearch_1    | # Possible solutions:
elasticsearch_1    | #   Reduce memory load on the system
elasticsearch_1    | #   Increase physical memory or swap space
elasticsearch_1    | #   Check if swap backing store is full
elasticsearch_1    | #   Use 64 bit Java on a 64 bit OS
elasticsearch_1    | #   Decrease Java heap size (-Xmx/-Xms)
elasticsearch_1    | #   Decrease number of Java threads
elasticsearch_1    | #   Decrease Java thread stack sizes (-Xss)
elasticsearch_1    | #   Set larger code cache with -XX:ReservedCodeCacheSize=
elasticsearch_1    | # This output file may be truncated or incomplete.
elasticsearch_1    | #
elasticsearch_1    | #  Out of Memory Error (os_linux.cpp:2658), pid=1, tid=0x00007f1ca7623ae8
elasticsearch_1    | #
elasticsearch_1    | # JRE version:  (8.0_131-b11) (build )
elasticsearch_1    | # Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
elasticsearch_1    | # Derivative: IcedTea 3.4.0
elasticsearch_1    | # Distribution: Custom build (Fri Jun 16 13:41:54 GMT 2017)
elasticsearch_1    | # Core dump written. Default location: /usr/share/elasticsearch/core or core.1
elasticsearch_1    | #
elasticsearch_1    |
elasticsearch_1    | ---------------  T H R E A D  ---------------
elasticsearch_1    |
elasticsearch_1    | Current thread (0x000055b58557d800):  JavaThread "Unknown thread" [_thread_in_vm, id=33, stack(0x00007f1ca7523000,0x00007f1ca7623aa8)]
elasticsearch_1    |
elasticsearch_1    | Stack: [0x00007f1ca7523000,0x00007f1ca7623aa8]
elasticsearch_1    |
elasticsearch_1    | ---------------  P R O C E S S  ---------------
elasticsearch_1    |
elasticsearch_1    | Java Threads: ( => current thread )
elasticsearch_1    |
elasticsearch_1    | Other Threads:
elasticsearch_1    |
elasticsearch_1    | =>0x000055b58557d800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=33, stack(0x00007f1ca7523000,0x00007f1ca7623aa8)]
elasticsearch_1    |
elasticsearch_1    | VM state:not at safepoint (not fully initialized)
elasticsearch_1    |
elasticsearch_1    | VM Mutex/Monitor currently owned by a thread: None
elasticsearch_1    |
elasticsearch_1    | GC Heap History (0 events):
elasticsearch_1    | No events
elasticsearch_1    |
elasticsearch_1    | Deoptimization events (0 events):
elasticsearch_1    | No events
elasticsearch_1    |
elasticsearch_1    | Internal exceptions (0 events):
elasticsearch_1    | No events
elasticsearch_1    |
elasticsearch_1    | Events (0 events):
elasticsearch_1    | No events
elasticsearch_1    |
elasticsearch_1    |
elasticsearch_1    | Dynamic libraries:
elasticsearch_1    | c0000000-c5330000 rw-p 00000000 00:00 0
elasticsearch_1    | 55b583d86000-55b583d87000 r-xp 00000000 ca:01 551288                     /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
elasticsearch_1    | 55b583f86000-55b583f87000 r--p 00000000 ca:01 551288                     /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
elasticsearch_1    | 55b583f87000-55b583f88000 rw-p 00001000 ca:01 551288                     /usr/lib/jvm/java-1.8-openjdk/jre/bin/java
elasticsearch_1    | 55b585565000-55b585594000 rw-p 00000000 00:00 0                          [heap]
elasticsearch_1    | 7f1c96104000-7f1c9622f000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1c9622f000-7f1c96405000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1c96405000-7f1c96406000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1c96406000-7f1c96407000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1c96407000-7f1c96512000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1c96512000-7f1c968c8000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1c968c8000-7f1c96b38000 rwxp 00000000 00:00 0
elasticsearch_1    | 7f1c96b38000-7f1ca58c8000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1ca58c8000-7f1ca5acf000 r-xp 00000000 ca:01 551337                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
elasticsearch_1    | 7f1ca5acf000-7f1ca5ad0000 r--p 00007000 ca:01 551337                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
elasticsearch_1    | 7f1ca5ad0000-7f1ca5ad1000 rw-p 00008000 ca:01 551337                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libzip.so
elasticsearch_1    | 7f1ca5ad1000-7f1ca5cfd000 r-xp 00000000 ca:01 551318                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
elasticsearch_1    | 7f1ca5cfd000-7f1ca5cfe000 r--p 0002c000 ca:01 551318                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
elasticsearch_1    | 7f1ca5cfe000-7f1ca5cff000 rw-p 0002d000 ca:01 551318                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libjava.so
elasticsearch_1    | 7f1ca5cff000-7f1ca5d00000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1ca5d00000-7f1ca5f0c000 r-xp 00000000 ca:01 551336                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
elasticsearch_1    | 7f1ca5f0c000-7f1ca5f0e000 r--p 0000c000 ca:01 551336                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
elasticsearch_1    | 7f1ca5f0e000-7f1ca5f0f000 rw-p 0000e000 ca:01 551336                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/libverify.so
elasticsearch_1    | 7f1ca5f0f000-7f1ca611f000 r-xp 00000000 ca:01 551451                     /usr/lib/libgcc_s.so.1
elasticsearch_1    | 7f1ca611f000-7f1ca6120000 r--p 00010000 ca:01 551451                     /usr/lib/libgcc_s.so.1
elasticsearch_1    | 7f1ca6120000-7f1ca6121000 rw-p 00011000 ca:01 551451                     /usr/lib/libgcc_s.so.1
elasticsearch_1    | 7f1ca6121000-7f1ca6461000 r-xp 00000000 ca:01 551509                     /usr/lib/libstdc++.so.6.0.22
elasticsearch_1    | 7f1ca6461000-7f1ca646f000 r--p 00140000 ca:01 551509                     /usr/lib/libstdc++.so.6.0.22
elasticsearch_1    | 7f1ca646f000-7f1ca6470000 rw-p 0014e000 ca:01 551509                     /usr/lib/libstdc++.so.6.0.22
elasticsearch_1    | 7f1ca6470000-7f1ca6473000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1ca6473000-7f1ca6e8a000 r-xp 00000000 ca:01 551341                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
elasticsearch_1    | 7f1ca6e8a000-7f1ca6f1f000 r--p 00817000 ca:01 551341                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
elasticsearch_1    | 7f1ca6f1f000-7f1ca6f47000 rw-p 008ac000 ca:01 551341                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so
elasticsearch_1    | 7f1ca6f47000-7f1ca6f77000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1ca6f77000-7f1ca718c000 r-xp 00000000 ca:01 550961                     /lib/libz.so.1.2.11
elasticsearch_1    | 7f1ca718c000-7f1ca718d000 r--p 00015000 ca:01 550961                     /lib/libz.so.1.2.11
elasticsearch_1    | 7f1ca718d000-7f1ca718e000 rw-p 00016000 ca:01 550961                     /lib/libz.so.1.2.11
elasticsearch_1    | 7f1ca718e000-7f1ca739a000 r-xp 00000000 ca:01 551302                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
elasticsearch_1    | 7f1ca739a000-7f1ca739b000 r--p 0000c000 ca:01 551302                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
elasticsearch_1    | 7f1ca739b000-7f1ca739c000 rw-p 0000d000 ca:01 551302                     /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/jli/libjli.so
elasticsearch_1    | 7f1ca739c000-7f1ca7425000 r-xp 00000000 ca:01 550954                     /lib/ld-musl-x86_64.so.1
elasticsearch_1    | 7f1ca7517000-7f1ca751f000 rw-s 00000000 ca:01 257070                     /tmp/hsperfdata_elasticsearch/1
elasticsearch_1    | 7f1ca751f000-7f1ca7520000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1ca7520000-7f1ca7521000 r--p 00000000 00:00 0
elasticsearch_1    | 7f1ca7521000-7f1ca7523000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1ca7523000-7f1ca7526000 ---p 00000000 00:00 0
elasticsearch_1    | 7f1ca7526000-7f1ca7624000 rw-p 00000000 00:00 0
elasticsearch_1    | 7f1ca7624000-7f1ca7625000 r--p 00088000 ca:01 550954                     /lib/ld-musl-x86_64.so.1
elasticsearch_1    | 7f1ca7625000-7f1ca7626000 rw-p 00089000 ca:01 550954                     /lib/ld-musl-x86_64.so.1
elasticsearch_1    | 7f1ca7626000-7f1ca7629000 rw-p 00000000 00:00 0
elasticsearch_1    | 7ffce82ac000-7ffce82cd000 rw-p 00000000 00:00 0                          [stack]
elasticsearch_1    | 7ffce8322000-7ffce8324000 r--p 00000000 00:00 0                          [vvar]
elasticsearch_1    | 7ffce8324000-7ffce8326000 r-xp 00000000 00:00 0                          [vdso]
elasticsearch_1    | ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
elasticsearch_1    |
elasticsearch_1    | VM Arguments:
elasticsearch_1    | jvm_args: -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Xms1g -Xmx1g -Des.path.home=/usr/share/elasticsearch
elasticsearch_1    | java_command: org.elasticsearch.bootstrap.Elasticsearch
elasticsearch_1    | java_class_path (initial): /usr/share/elasticsearch/lib/lucene-analyzers-common-6.6.1.jar:/usr/share/elasticsearch/lib/elasticsearch-5.6.4.jar:/usr/share/elasticsearch/lib/hppc-0.7.1.jar:/usr/share/elasticsearch/lib/lucene-grouping-6.6.1.jar:/usr/share/elasticsearch/lib/lucene-misc-6.6.1.jar:/usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.6.jar:/usr/share/elasticsearch/lib/log4j-1.2-api-2.9.1.jar:/usr/share/elasticsearch/lib/lucene-memory-6.6.1.jar:/usr/share/elasticsearch/lib/lucene-queryparser-6.6.1.jar:/usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.6.jar:/usr/share/elasticsearch/lib/lucene-suggest-6.6.1.jar:/usr/share/elasticsearch/lib/jopt-simple-5.0.2.jar:/usr/share/elasticsearch/lib/jackson-core-2.8.6.jar:/usr/share/elasticsearch/lib/joda-time-2.9.5.jar:/usr/share/elasticsearch/lib/spatial4j-0.6.jar:/usr/share/elasticsearch/lib/jna-4.4.0-1.jar:/usr/share/elasticsearch/lib/HdrHistogram-2.1.9.jar:/usr/share/elasticsearch/lib/lucene-spatial-6.6.1.jar:/usr/share/elasticsearch/lib/plugin-cli-5.6.4.jar:/usr/share/elasticsearch/lib/lucene-spatial-extras-6.6.1.jar:/usr/share/elasticsearch/lib/lucene-join-6.6.1.jar:/usr/share/elasticsearch/lib/log4j-api-2.9.1.jar:/usr/share/elasticsearch/lib/lucene-sandbox-6.6.1.jar:/usr/share/elasticsearch/lib/t-digest-3.0.jar:/usr/share/elasticsearch/lib/jts-1.13.jar:/usr/share/elasticsearch/lib/lucene-spatial3d-6.6.1.jar:/usr/share/elasticsearch/lib/java-version-checker-5.6.4.jar:/usr/share/elasticsearch/lib/lucene-backward-codecs-6.6.1.jar:/usr/share/elasticsearch/lib/snakeyaml-1.15.jar:/usr/share/elasticsearch/lib/securesm-1.1.jar:/usr/share/elasticsearch/lib/lucene-core-6.6.1.jar:/usr/share/elasticsearch/lib/lucene-highlighter-6.6.1.jar:/usr/share/elasticsearch/lib/log4j-core-2.9.1.jar:/usr/share/elasticsearch/lib/lucene-queries-6.6.1.jar:/usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.6.jar
elasticsearch_1    | Launcher Type: SUN_STANDARD
elasticsearch_1    |
elasticsearch_1    | Environment Variables:
elasticsearch_1    | JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk/jre
elasticsearch_1    | PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
elasticsearch_1    | LD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64
elasticsearch_1    |
elasticsearch_1    | Signal Handlers:
elasticsearch_1    | SIGSEGV: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGBUS: [libjvm.so+0x63094c], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGFPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGPIPE: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGXFSZ: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGILL: [libjvm.so+0x55ffac], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
elasticsearch_1    | SIGUSR2: [libjvm.so+0x56014b], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
elasticsearch_1    | SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
elasticsearch_1    | SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
elasticsearch_1    | SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
elasticsearch_1    | SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
elasticsearch_1    |
elasticsearch_1    |
elasticsearch_1    | ---------------  S Y S T E M  ---------------
elasticsearch_1    |
elasticsearch_1    | OS:NAME="Alpine Linux"
elasticsearch_1    | ID=alpine
elasticsearch_1    | VERSION_ID=3.6.2
elasticsearch_1    | PRETTY_NAME="Alpine Linux v3.6"
elasticsearch_1    | HOME_URL="http://alpinelinux.org"
elasticsearch_1    | BUG_REPORT_URL="http://bugs.alpinelinux.org"
elasticsearch_1    |
elasticsearch_1    | uname:Linux 4.4.0-1041-aws #50-Ubuntu SMP Wed Nov 15 22:18:17 UTC 2017 x86_64
elasticsearch_1    | libc:glibc 2.9 NPTL
elasticsearch_1    | rlimit: STACK 8192k, CORE infinity, NPROC infinity, NOFILE 1048576, AS infinity
elasticsearch_1    | load average:0.09 0.21 0.09
elasticsearch_1    |
elasticsearch_1    | /proc/meminfo:
elasticsearch_1    | MemTotal:        1014648 kB
elasticsearch_1    | MemFree:          360592 kB
elasticsearch_1    | MemAvailable:     464424 kB
elasticsearch_1    | Buffers:           21656 kB
elasticsearch_1    | Cached:           214312 kB
elasticsearch_1    | SwapCached:        35112 kB
elasticsearch_1    | Active:           332812 kB
elasticsearch_1    | Inactive:         216500 kB
elasticsearch_1    | Active(anon):     224040 kB
elasticsearch_1    | Inactive(anon):   112780 kB
elasticsearch_1    | Active(file):     108772 kB
elasticsearch_1    | Inactive(file):   103720 kB
elasticsearch_1    | Unevictable:        3660 kB
elasticsearch_1    | Mlocked:            3660 kB
elasticsearch_1    | SwapTotal:        524284 kB
elasticsearch_1    | SwapFree:         310444 kB
elasticsearch_1    | Dirty:             18968 kB
elasticsearch_1    | Writeback:             0 kB
elasticsearch_1    | AnonPages:        294304 kB
elasticsearch_1    | Mapped:           105936 kB
elasticsearch_1    | Shmem:             21036 kB
elasticsearch_1    | Slab:              64204 kB
elasticsearch_1    | SReclaimable:      38124 kB
elasticsearch_1    | SUnreclaim:        26080 kB
elasticsearch_1    | KernelStack:        5776 kB
elasticsearch_1    | PageTables:        15088 kB
elasticsearch_1    | NFS_Unstable:          0 kB
elasticsearch_1    | Bounce:                0 kB
elasticsearch_1    | WritebackTmp:          0 kB
elasticsearch_1    | CommitLimit:     1031608 kB
elasticsearch_1    | Committed_AS:    2267284 kB
elasticsearch_1    | VmallocTotal:   34359738367 kB
elasticsearch_1    | VmallocUsed:           0 kB
elasticsearch_1    | VmallocChunk:          0 kB
elasticsearch_1    | HardwareCorrupted:     0 kB
elasticsearch_1    | AnonHugePages:    120832 kB
elasticsearch_1    | CmaTotal:              0 kB
elasticsearch_1    | CmaFree:               0 kB
elasticsearch_1    | HugePages_Total:       0
elasticsearch_1    | HugePages_Free:        0
elasticsearch_1    | HugePages_Rsvd:        0
elasticsearch_1    | HugePages_Surp:        0
elasticsearch_1    | Hugepagesize:       2048 kB
elasticsearch_1    | DirectMap4k:       81920 kB
elasticsearch_1    | DirectMap2M:      966656 kB
elasticsearch_1    |
elasticsearch_1    |
elasticsearch_1    | CPU:total 1 (initial active 1) (1 cores per cpu, 1 threads per core) family 6 model 63 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, tsc, bmi1, bmi2
elasticsearch_1    |
elasticsearch_1    | /proc/cpuinfo:
elasticsearch_1    | processor  : 0
elasticsearch_1    | vendor_id  : GenuineIntel
elasticsearch_1    | cpu family : 6
elasticsearch_1    | model              : 63
elasticsearch_1    | model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
elasticsearch_1    | stepping   : 2
elasticsearch_1    | microcode  : 0x3a
elasticsearch_1    | cpu MHz            : 2400.052
elasticsearch_1    | cache size : 30720 KB
elasticsearch_1    | physical id        : 0
elasticsearch_1    | siblings   : 1
elasticsearch_1    | core id            : 0
elasticsearch_1    | cpu cores  : 1
elasticsearch_1    | apicid             : 0
elasticsearch_1    | initial apicid     : 0
elasticsearch_1    | fpu                : yes
elasticsearch_1    | fpu_exception      : yes
elasticsearch_1    | cpuid level        : 13
elasticsearch_1    | wp         : yes
elasticsearch_1    | flags              : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt
elasticsearch_1    | bugs               :
elasticsearch_1    | bogomips   : 4800.10
elasticsearch_1    | clflush size       : 64
elasticsearch_1    | cache_alignment    : 64
elasticsearch_1    | address sizes      : 46 bits physical, 48 bits virtual
elasticsearch_1    | power management:
elasticsearch_1    |
elasticsearch_1    |
elasticsearch_1    |
elasticsearch_1    | Memory: 4k page, physical 1014648k(360592k free), swap 524284k(310444k free)
elasticsearch_1    |
elasticsearch_1    | vm_info: OpenJDK 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Jun 16 2017 13:51:29 by "buildozer" with gcc 6.3.0
elasticsearch_1    |
elasticsearch_1    | time: Sun Dec  3 11:13:59 2017
elasticsearch_1    | elapsed time: 0 seconds (0d 0h 0m 0s)
elasticsearch_1    |
awselkbilling_elasticsearch_1 exited with code 1
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen t
o 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","plugin:kibana@5.1.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","plugin:elasticsearch@5.1.1","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["error","elasticsearch"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => getaddrinfo ENOTFOUND elasticsearch elasticsearch:9200"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","plugin:elasticsearch@5.1.1","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","plugin:console@5.1.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","plugin:timelion@5.1.1","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["listening","info"],"pid":1,"message":"Server running at http://0.0.0.0:5601"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:07Z","tags":["status","ui settings","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Elasticsearch plugin is red","prevState":"uninitialized","prevMsg":"uninitialized"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:09Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:09Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:12Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:12Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:14Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:14Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:17Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:17Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:19Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:19Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:22Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:22Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1         | 11:14:22.709 [main] INFO  logstash.modules.scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
logstash_1         | 11:14:22.720 [main] INFO  logstash.modules.scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
logstash_1         | 11:14:23.882 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1         | 11:14:23.883 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:24.096 [[main]-pipeline-manager] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
logstash_1         | 11:14:24.102 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Using mapping template from {:path=>nil}
logstash_1         | 11:14:24.110 [[main]-pipeline-manager] ERROR logstash.outputs.elasticsearch - Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:37:in `read_template_file'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:23:in `get_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:58:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/common.rb:25:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:in `start_pipeline'"]}
logstash_1         | 11:14:24.122 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1         | 11:14:24.230 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
logstash_1         | 11:14:24.249 [[main]-pipeline-manager] INFO  logstash.inputs.tcp - Automatically switching from json to json_lines codec {:plugin=>"tcp"}
logstash_1         | 11:14:24.644 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
logstash_1         | 11:14:24.750 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:24Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:24Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:27Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:27Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:14:29.186 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:29.196 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:29Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:29Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:32Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:32Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
logstash_1         | 11:14:34.198 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:34.205 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:34Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:34Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:37Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:37Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:14:39.209 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:39.218 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:39Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:39Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:42Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:42Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:14:44.219 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:44.225 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:44Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:44Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:47Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:47Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:14:49.227 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:49.235 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:49Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:49Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:52Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:52Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
logstash_1         | 11:14:54.236 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:54.244 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:55Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:55Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Elasticsearch container has started to listen to 9200
aws-elk-billing_1  | Something's wrong with Elasticsearch. Exception is [Errno -2] Name or service not known
aws-elk-billing_1  | I will retry after 4 seconds
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:57Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:14:57Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:14:59.245 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:14:59.253 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:00Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:00Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
aws-elk-billing_1  | Checking if Logstash container has started to listen to 5140
aws-elk-billing_1  | Great Logstash is listening on 5140 :)
aws-elk-billing_1  | Checking if Kibana container has started to listen to 5601
aws-elk-billing_1  | Great Kibana is listening on 5601 :)
aws-elk-billing_1  | Traceback (most recent call last):
aws-elk-billing_1  |   File "/aws-elk-billing/orchestrate.py", line 24, in <module>
aws-elk-billing_1  |     tools.index_template()
aws-elk-billing_1  |   File "/aws-elk-billing/tools/tools.py", line 80, in index_template
aws-elk-billing_1  |     out = subprocess.check_output(['curl --head "elasticsearch:9200/_template/aws_billing"'], shell=True, stderr=subprocess.PIPE)
aws-elk-billing_1  |   File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
aws-elk-billing_1  |     raise CalledProcessError(retcode, cmd, output=output)
aws-elk-billing_1  | subprocess.CalledProcessError: Command '['curl --head "elasticsearch:9200/_template/aws_billing"']' returned non-zero exit status 6
awselkbilling_aws-elk-billing_1 exited with code 1
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:02Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:02Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:04.254 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:15:04.259 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:05Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:05Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:07Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:07Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:09.261 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:15:09.269 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:10Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:10Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:12Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:12Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:14.270 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:15:14.277 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:15Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:15Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:17Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:17Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:19.279 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:15:19.286 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:20Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:20Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:22Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:22Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:24.288 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1         | 11:15:24.293 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name does not resolve"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:25Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:25Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:27Z","tags":["warning","elasticsearch"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1           | {"type":"log","@timestamp":"2017-12-03T11:15:27Z","tags":["warning","elasticsearch"],"pid":1,"message":"No living connections"}
logstash_1         | 11:15:29.294 [Ruby-0-Thread-6: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.4.2-java/lib/logstash/o
droidlabour commented 6 years ago

@kenkit Elasticsearch container is stopping due to insufficient memory.

kenkit commented 6 years ago

I tried changing it in docker-compose.yml but it doesn't seem to affect anything, I still see java is called with 1gb as default.

droidlabour commented 6 years ago

@kenkit You need to check the actual heap size in the logs. It should read something like this [pNnt_i6] heap size [1007.3mb], compressed ordinary object pointers [true] Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

kenkit commented 6 years ago

I fixed it. Thank you