Multiple pipelines conflict.

[schivei]

When I try to input an unique collection from mongo to elasticsearch, it runs correctly, but if I try to reach many collections (in different pipelines) to put each in many indices, all indices are populatated with same documents.

I'm using logstash 6.3.0 and each pipeline is configured with different SQLite db file.

While pipeline worker is running, all SQLite db file has no data populated.

Logstash: 6.3.0 Elasticsearch: 6.3.0 OS: ubuntu 16.04 Memory: 16GB Processors: 2

[schivei]

SQLite db files

2.0K Sep 11 12:17 logstash_sqlite_a.db
2.0K Sep 11 12:17 logstash_sqlite_b.db
2.0K Sep 11 12:17 logstash_sqlite_c.db
2.0K Sep 11 12:17 logstash_sqlite_d.db
2.0K Sep 11 12:17 logstash_sqlite_e.db
2.0K Sep 11 12:17 logstash_sqlite_f.db
2.0K Sep 11 12:17 logstash_sqlite_g.db

First two pipelines example

input {
        mongodb {
                uri => 'mongodb://192.168.0.123:27300/BigData'
                placeholder_db_dir => '/opt/logstash-mongodb'
                placeholder_db_name => 'logstash_sqlite_a.db'
                collection => '^(first_collection)$'
                generateId => true
                batch_size => 1000
        }
}

filter {
  mutate {
    rename => { "_id" => "mongo_id" }
  }
}

output {
        elasticsearch {
                index => "first-collection"
                document_type => "docs"
                doc_as_upsert => true
                document_id => "%{mongo_id}"
        }
}

input {
        mongodb {
                uri => 'mongodb://192.168.0.123:27300/BigData'
                placeholder_db_dir => '/opt/logstash-mongodb'
                placeholder_db_name => 'logstash_sqlite_b.db'
                collection => '^(second_collection)$'
                generateId => true
                batch_size => 1000
        }
}

filter {
  mutate {
    rename => { "_id" => "mongo_id" }
  }
}

output {
        elasticsearch {
                index => "second-collection"
                document_type => "docs"
                doc_as_upsert => true
                document_id => "%{mongo_id}"
        }
}

I have 7 pipelines like this two.

All resultant indices have same docs.

index                   docs.count
first-collection        340595
second-collection       340595
third-collection        340595
fourth-collection       340595
fifth-collection        340595
sixth-collection        340595
seventh-collection      340595