search

deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.27k stars 6.79k forks source link

All indexes are populated with data from all resources #887

Closed rider4 closed 1 year ago

rider4 commented 1 year ago

Problem description

As you see below, we extended only one new port 5045 for docker-compose. We have only one producer of data that pushes data to port 4045 and should be handled with logstash_45.conf. We created a fake config logstash_46.conf. This pipeline can't listen to data as a minimum because port 4046 is not extended with the docker container. The problem is that data received from a server through port 4045 populates to both indexes microservice-1/microservice-2 and at the same time. The main idea is very simple - to split data from a few servers to their own indexes.

Extra information

Stack configuration

 $ git status
On branch release-7.x
Your branch is up to date with 'origin/release-7.x'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   docker-compose.yml
        deleted:    logstash/pipeline/logstash.conf

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        logstash/pipeline/logstash_45.conf
        logstash/pipeline/logstash_46.conf
diff --git a/docker-compose.yml b/docker-compose.yml
index 5e87988..a22a321 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -77,6 +77,7 @@ services:
       - ./logstash/pipeline:/usr/share/logstash/pipeline:ro,Z
     ports:
       - 5044:5044
+      - 5045:5045
       - 50000:50000/tcp
       - 50000:50000/udp
       - 9600:9600

$ cat logstash/pipeline/logstash_45.conf 
input {
  beats {
    port => 5045
  }
}

filter {
  date {
   match => [ "time", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" ]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "elastic"
    password => "changeme"
    index => "microservice-1"
  }
}
$ cat logstash/pipeline/logstash_46.conf 
input {
  beats {
    port => 5046
  }
}

filter {
  date {
   match => [ "time", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" ]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "elastic"
    password => "changeme"
    index => "microservice-2"
  }
}

Docker setup

$ docker version

Client: Docker Engine - Community
 Version:           24.0.4
 API version:       1.43
 Go version:        go1.20.5
 Git commit:        3713ee1
 Built:             Fri Jul  7 14:50:55 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.4
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.5
  Git commit:       4ffc614
  Built:            Fri Jul  7 14:50:55 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker-compose version

docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.10.6
OpenSSL version: OpenSSL 3.0.2 15 Mar 2022

Container logs

$ docker-compose logs

logstash_1       | 2023-07-12T15:12:49.414358438Z Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1       | 2023-07-12T15:12:49.471472470Z [2023-07-12T15:12:49,469][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash_1       | 2023-07-12T15:12:49.478946073Z [2023-07-12T15:12:49,478][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.11", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.19+7 on 11.0.19+7 +indy +jit [linux-x86_64]"}
logstash_1       | 2023-07-12T15:12:49.481541324Z [2023-07-12T15:12:49,481][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xms256m, -Xmx256m]
logstash_1       | 2023-07-12T15:12:50.700812147Z [2023-07-12T15:12:50,698][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash_1       | 2023-07-12T15:12:51.646860701Z [2023-07-12T15:12:51,646][INFO ][org.reflections.Reflections] Reflections took 58 ms to scan 1 urls, producing 119 keys and 419 values 
logstash_1       | 2023-07-12T15:12:52.279236581Z [2023-07-12T15:12:52,278][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.307529204Z [2023-07-12T15:12:52,307][WARN ][deprecation.logstash.inputs.beats] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.330454655Z [2023-07-12T15:12:52,330][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.335658110Z [2023-07-12T15:12:52,335][WARN ][deprecation.logstash.inputs.beats] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.392396555Z [2023-07-12T15:12:52,392][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.419373936Z [2023-07-12T15:12:52,419][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.445242320Z [2023-07-12T15:12:52,445][WARN ][deprecation.logstash.codecs.plain] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.456669529Z [2023-07-12T15:12:52,456][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:12:52.499862633Z [2023-07-12T15:12:52,498][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1       | 2023-07-12T15:12:52.724263504Z [2023-07-12T15:12:52,719][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1       | 2023-07-12T15:12:52.924282996Z [2023-07-12T15:12:52,923][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1       | 2023-07-12T15:12:52.935242726Z [2023-07-12T15:12:52,934][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.11) {:es_version=>7}
logstash_1       | 2023-07-12T15:12:52.936999500Z [2023-07-12T15:12:52,936][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1       | 2023-07-12T15:12:52.989009503Z [2023-07-12T15:12:52,986][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
logstash_1       | 2023-07-12T15:12:52.989590035Z [2023-07-12T15:12:52,989][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
logstash_1       | 2023-07-12T15:12:52.991949061Z [2023-07-12T15:12:52,991][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1       | 2023-07-12T15:12:53.001282252Z [2023-07-12T15:12:53,000][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@elasticsearch:9200/]}}
logstash_1       | 2023-07-12T15:12:53.012556182Z [2023-07-12T15:12:53,012][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@elasticsearch:9200/"}
logstash_1       | 2023-07-12T15:12:53.018643620Z [2023-07-12T15:12:53,018][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (7.17.11) {:es_version=>7}
logstash_1       | 2023-07-12T15:12:53.018977398Z [2023-07-12T15:12:53,018][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
logstash_1       | 2023-07-12T15:12:53.039400292Z [2023-07-12T15:12:53,039][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
logstash_1       | 2023-07-12T15:12:53.048411848Z [2023-07-12T15:12:53,039][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
logstash_1       | 2023-07-12T15:12:53.062345410Z [2023-07-12T15:12:53,061][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
logstash_1       | 2023-07-12T15:12:53.116692529Z [2023-07-12T15:12:53,115][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash_45.conf", "/usr/share/logstash/pipeline/logstash_46.conf"], :thread=>"#<Thread:0x6444fc86 run>"}
logstash_1       | 2023-07-12T15:12:53.788764926Z [2023-07-12T15:12:53,788][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.67}
logstash_1       | 2023-07-12T15:12:53.808869744Z [2023-07-12T15:12:53,808][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5046"}
logstash_1       | 2023-07-12T15:12:53.816973663Z [2023-07-12T15:12:53,816][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5045"}
logstash_1       | 2023-07-12T15:12:53.820853749Z [2023-07-12T15:12:53,820][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1       | 2023-07-12T15:12:53.920741385Z [2023-07-12T15:12:53,913][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1       | 2023-07-12T15:12:53.977975282Z [2023-07-12T15:12:53,977][INFO ][org.logstash.beats.Server][main][48187f8f48f8827d3394b8b6e9e1b3384451f9f3959abc4fbbc963ac13d26f86] Starting server on port: 5045
logstash_1       | 2023-07-12T15:12:53.990890750Z [2023-07-12T15:12:53,990][INFO ][org.logstash.beats.Server][main][50b9c9c24f4e31780bc8cbe32e25f35c3a334e2e7b4d6e219570f17688d8d3e1] Starting server on port: 5046
logstash_1       | 2023-07-12T15:12:59.431617066Z [2023-07-12T15:12:59,431][WARN ][deprecation.logstash.codecs.plain][main][48187f8f48f8827d3394b8b6e9e1b3384451f9f3959abc4fbbc963ac13d26f86] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1       | 2023-07-12T15:14:15.581803364Z [2023-07-12T15:14:15,581][WARN ][deprecation.logstash.codecs.plain][main][48187f8f48f8827d3394b8b6e9e1b3384451f9f3959abc4fbbc963ac13d26f86] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
antoineco commented 1 year ago

This question is about the usage of Elastic products, not the Docker setup provided by docker-elk or its default configuration.

I recommend checking the documentation at https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html. You're currently using a single pipeline, for which all files under pipeline/* get merged.

rider4 commented 1 year ago

Thank a lot.