How to test dead letter queue in Logstash

[TomonoriSoejima]

1. create a mapping dynamically with message as number type

POST number/_doc/
{
  "message" : 4
}

2. add dead_letter_queue.enable: true to logstash.yml

3. make a conf file with stdin in input.

input {
  stdin { }
}

output {
  stdout {
    codec => rubydebug
  }

elasticsearch {
    hosts => "localhost:9200"
    index => number
    }
}

4. insert string like test after starting logstash.

That should internally cause a mapping conflict error and you should see it in the dead_letter_queue directory.

[surfer-3:data]$ find dead_letter_queue/
dead_letter_queue/
dead_letter_queue//main
dead_letter_queue//main/.lock
dead_letter_queue//main/2.log
dead_letter_queue//main/3.log
dead_letter_queue//main/1.log
dead_letter_queue//main/4.log.tmp
[surfer-3:data]$ cat dead_letter_queue//main/3.log
1cn???????023-06-14T01:33:30.477588Zo?qjava.util.HashMap?dDATA?xorg.logstash.ConvertedMap?eevent?xorg.logstash.ConvertedMap?horiginal?torg.jruby.RubyStringfhello2???j@timestamp?vorg.logstash.Timestampx023-06-14T01:33:30.325762Z?dhost?xorg.logstash.ConvertedMap?hhostname?torg.jruby.RubyStringnsurfer-3.local???gmessage?torg.jruby.RubyStringfhello2?h@versiona1??dMETA?xorg.logstash.Convertedelasticsearch?Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"number", :routing=>nil}, {"event"=>{"original"=>"hello2"}, "@timestamp"=>2023-06-14T01:33:30.325762Z, "host"=>{"hostname"=>"surfer-3.local"}, "message"=>"hello2", "@version"=>"1"}], response: {"index"=>{"_index"=>"number", "_id"=>"mrqJt4gBWu69v_6CU6jm", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [message] of type [long] in document with id 'mrqJt4gBWu69v_6CU6jm'. Preview of field's value: 'hello2'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"hello2\""}}}}[surfer-3:data]$