robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 592 forks source link

kubana still telling me (Kibana server is not ready yet) #417

Closed ghost closed 4 years ago

ghost commented 4 years ago

My docker compose the ports extra defined to open it for external networks

version: '3'

services:
  elastiflow-elasticsearch-oss:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.3.0
    container_name: elastiflow-elasticsearch-oss
    restart: 'no'
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /var/lib/elastiflow_es:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    environment:
      # JVM Heap size
      #   - this should be at least 2GB for simple testing, receiving only a few flows per second.
      #   - for production environments upto 31GB is recommended.
      ES_JAVA_OPTS: '-Xms3g -Xmx3g'

      cluster.name: elastiflow

      bootstrap.memory_lock: 'true'

      network.host: "0.0.0.0"
      http.port: 9200
      discovery.type: 'single-node'

      indices.query.bool.max_clause_count: 8192
      search.max_buckets: 100000

      action.destructive_requires_name: 'true'

  elastiflow-kibana-oss:
    image: docker.elastic.co/kibana/kibana-oss:7.3.0
    container_name: elastiflow-kibana-oss
    restart: 'no'
    depends_on:
      - elastiflow-elasticsearch-oss
    ports:
      - "5601:5601"
    environment:
      SERVER_HOST: "0.0.0.0"
      SERVER_PORT: 5601

      ELASTICSEARCH_HOSTS: "http://10.0.30.29:9200"

      KIBANA_DEFAULTAPPID: "dashboard/653cf1e0-2fd2-11e7-99ed-49759aed30f5"

      LOGGING_QUIET: 'true'

  elastiflow-logstash-oss:
    image: robcowart/elastiflow-logstash-oss:3.5.1
    container_name: elastiflow-logstash-oss
    restart: 'no'
    ports:
      - "2055:2055"
      - "6343:6343"
      - "4739:4739"
    depends_on:
      - elastiflow-elasticsearch-oss
    environment:
      # JVM Heap size - this MUST be at least 3GB (4GB preferred)
      LS_JAVA_OPTS: '-Xms5g -Xmx5g'
      ELASTIFLOW_ES_HOST: "172.19.0.2"
      # ElastiFlow global configuration
      ELASTIFLOW_DEFAULT_APPID_SRCTYPE: "__UNKNOWN"

      # Name resolution option
      ELASTIFLOW_RESOLVE_IP2HOST: "false"
      ELASTIFLOW_NAMESERVER: "127.0.0.1"

      ELASTIFLOW_NETFLOW_IPV4_PORT: 2055
      ELASTIFLOW_SFLOW_IPV4_PORT: 6343
      ELASTIFLOW_IPFIX_TCP_IPV:4_PORT: 4739
docker-compose up
Starting elastiflow-elasticsearch-oss ... done
Starting elastiflow-logstash-oss      ... done
Starting elastiflow-kibana-oss        ... done
Attaching to elastiflow-elasticsearch-oss, elastiflow-kibana-oss, elastiflow-logstash-oss
elastiflow-elasticsearch-oss    | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
elastiflow-kibana-oss           | {"type":"log","@timestamp":"2019-09-22T10:13:16Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nGET http://10.0.30.29:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect EHOSTUNREACH 10.0.30.29:9200"}
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,188+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/fedora-root)]], net usable_space [216.8gb], net total_space [224.8gb], types [xfs]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,219+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "heap size [2.9gb], compressed ordinary object pointers [true]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,253+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "node name [918624de194a], node ID [HY3MmE8RSDWf2nrRLsZb-A], cluster name [elastiflow]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,264+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "version[7.3.0], pid[1], build[oss/docker/de777fa/2019-07-24T18:30:11.767338Z], OS[Linux/5.2.15-200.fc30.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,265+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "JVM home [/usr/share/elasticsearch/jdk]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:17,266+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-4150948664004315653, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xms3g, -Xmx3g, -Dio.netty.allocator.type=pooled, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker, -Des.bundled_jdk=true]"  }
elastiflow-kibana-oss           | {"type":"log","@timestamp":"2019-09-22T10:13:17Z","tags":["status","plugin:elasticsearch@7.3.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - No Living connections","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,034+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [aggs-matrix-stats]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,038+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [analysis-common]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,046+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [ingest-common]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,046+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [ingest-geoip]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,047+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [ingest-user-agent]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,050+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [lang-expression]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,051+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [lang-mustache]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,051+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [lang-painless]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,052+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [mapper-extras]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,052+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [parent-join]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,052+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [percolator]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,053+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [rank-eval]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,053+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [reindex]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,054+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [repository-url]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,054+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "loaded module [transport-netty4]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:20,055+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "no plugins loaded"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:31,988+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "using discovery type [single-node] and seed hosts providers [settings]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:33,744+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "initialized"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:33,745+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "starting ..."  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,364+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "publish_address {172.19.0.2:9300}, bound_addresses {[::]:9300}"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,414+0000", "level": "WARN", "component": "o.e.b.BootstrapChecks", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,433+0000", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "cluster UUID [rBiZpclMSoqQohx8mp77pg]"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,660+0000", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "elected-as-master ([1] nodes joined)[{918624de194a}{HY3MmE8RSDWf2nrRLsZb-A}{FdtGbk3jREmBombdKbiOJw}{172.19.0.2}{172.19.0.2:9300}{dim} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 12, version: 46, reason: master node changed {previous [], current [{918624de194a}{HY3MmE8RSDWf2nrRLsZb-A}{FdtGbk3jREmBombdKbiOJw}{172.19.0.2}{172.19.0.2:9300}{dim}]}"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,795+0000", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "elastiflow", "node.name": "918624de194a",  "message": "master node changed {previous [], current [{918624de194a}{HY3MmE8RSDWf2nrRLsZb-A}{FdtGbk3jREmBombdKbiOJw}{172.19.0.2}{172.19.0.2:9300}{dim}]}, term: 12, version: 46, reason: Publication{term=12, version=46}"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,886+0000", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elastiflow", "node.name": "918624de194a", "cluster.uuid": "rBiZpclMSoqQohx8mp77pg", "node.id": "HY3MmE8RSDWf2nrRLsZb-A",  "message": "publish_address {172.19.0.2:9200}, bound_addresses {[::]:9200}"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:34,888+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elastiflow", "node.name": "918624de194a", "cluster.uuid": "rBiZpclMSoqQohx8mp77pg", "node.id": "HY3MmE8RSDWf2nrRLsZb-A",  "message": "started"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:35,023+0000", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "elastiflow", "node.name": "918624de194a", "cluster.uuid": "rBiZpclMSoqQohx8mp77pg", "node.id": "HY3MmE8RSDWf2nrRLsZb-A",  "message": "recovered [1] indices into cluster_state"  }
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:13:36,131+0000", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "elastiflow", "node.name": "918624de194a", "cluster.uuid": "rBiZpclMSoqQohx8mp77pg", "node.id": "HY3MmE8RSDWf2nrRLsZb-A",  "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...])."  }
elastiflow-logstash-oss         | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
elastiflow-logstash-oss         | [2019-09-22T10:13:52,082][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
elastiflow-logstash-oss         | [2019-09-22T10:13:52,102][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
elastiflow-logstash-oss         | [2019-09-22T10:13:53,490][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.1.4"}
elastiflow-logstash-oss         | [2019-09-22T10:13:54,193][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
elastiflow-logstash-oss         | [2019-09-22T10:16:53,943][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@172.19.0.2:9200/]}}
elastiflow-logstash-oss         | [2019-09-22T10:16:54,515][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@172.19.0.2:9200/"}
elastiflow-logstash-oss         | [2019-09-22T10:16:54,877][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
elastiflow-logstash-oss         | [2019-09-22T10:16:54,886][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
elastiflow-logstash-oss         | [2019-09-22T10:16:54,974][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}
elastiflow-logstash-oss         | [2019-09-22T10:16:54,986][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.19.0.2"]}
elastiflow-logstash-oss         | [2019-09-22T10:16:55,378][INFO ][logstash.outputs.elasticsearch] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
elastiflow-logstash-oss         | [2019-09-22T10:16:55,386][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"order"=>0, "version"=>30501, "index_patterns"=>"elastiflow-3.5.1-*", "settings"=>{"index"=>{"number_of_shards"=>3,
..............................
keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_addr"=>{"type"=>"ip"}, "server_asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_country_code"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point"}, "server_hostname"=>{"type"=>"keyword"}, "server_rep_tags"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{"type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_country_code"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=>{"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}}
elastiflow-logstash-oss         | [2019-09-22T10:36:30,164][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.5.1
elastiflow-elasticsearch-oss    | {"type": "server", "timestamp": "2019-09-22T10:36:30,851+0000", "level": "INFO", "component": "o.e.c.m.MetaDataIndexTemplateService", "cluster.name": "elastiflow", "node.name": "bb915980b380", "cluster.uuid": "rBiZpclMSoqQohx8mp77pg", "node.id": "HY3MmE8RSDWf2nrRLsZb-A",  "message": "adding template [elastiflow-3.5.1] for index patterns [elastiflow-3.5.1-*]"  }
elastiflow-logstash-oss         | [2019-09-22T10:36:31,660][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
elastiflow-logstash-oss         | [2019-09-22T10:36:31,703][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
elastiflow-logstash-oss         | [2019-09-22T10:36:41,622][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
elastiflow-logstash-oss         | [2019-09-22T10:36:41,624][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
elastiflow-logstash-oss         | [2019-09-22T10:36:51,816][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x6ea87f71 run>"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,190][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"elastiflow"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,373][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,387][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:4739", :ssl_enable=>"false"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,421][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:4739"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,434][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:6343"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,541][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
elastiflow-logstash-oss         | [2019-09-22T10:36:52,544][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"2048"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,573][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
elastiflow-logstash-oss         | [2019-09-22T10:36:52,574][WARN ][logstash.inputs.udp      ] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 212992 bytes.
elastiflow-logstash-oss         | [2019-09-22T10:36:52,602][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"212992", :queue_size=>"2048"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,603][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"212992", :queue_size=>"2048"}
elastiflow-logstash-oss         | [2019-09-22T10:36:52,686][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["elastiflow"]}
bdonfouet commented 4 years ago

try using: ELASTIFLOW_ES_HOST: "elastiflow-elasticsearch-oss:" ELASTICSEARCH_HOSTS: "http://elastiflow-elasticsearch-oss::9200"

ghost commented 4 years ago

Thx but not working

elastiflow-logstash-oss | [2019-09-23T21:12:55,067][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@elastiflow-logstash-oss:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@elastiflow-logstash-oss:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

robcowart commented 4 years ago

You seem to have some issues setting up Docker networking. The provided docker-compose.yml uses host networking. Can you explain the intent of your changes?

Tarik-H commented 4 years ago

@elisa9180 If you're planning to run everything from the same compose file, you should point to your ES service: ELASTICSEARCH_HOSTS: "http://elastiflow-elasticsearch-oss:9200" ELASTIFLOW_ES_HOST: "elastiflow-elasticsearch-oss:9200"

robcowart commented 4 years ago

Closing due to no feedback. Please reopen if this is still an issue.