yaauie
commented
1 year ago Is this an artificial reproduction? Are we expecting the events that are rejected by your output to get re-injected into the same pipeline?
Closed ukho-cfreeman closed 1 year ago
yaauie
commented
1 year ago Is this an artificial reproduction? Are we expecting the events that are rejected by your output to get re-injected into the same pipeline?
ukho-cfreeman
commented
1 year ago Hi @yaauie
I didn't include the full configuration which hasn't helped, my apologies. When events get rejected they get re-ingested via the DLQ input plugin but are modified to include the DLQ reason and converted to a string.
To confirm this is happening in all live environments.
input {
kinesis {
kinesis_stream_name => "stream_name"
initial_position_in_stream => "LATEST"
application_name => "application_name"
region => "eu-west-2"
codec => json { ecs_compatibility => "disabled" }
additional_settings => {"initial_lease_table_read_capacity" => 20 "initial_lease_table_write_capacity" => 100}
}
}
input {
dead_letter_queue {
id => "kubernetes_dlq"
path => "/usr/share/logstash/data/dead_letter_queue"
sincedb_path => "/usr/share/logstash/data/sincedb_dlq"
commit_offsets => true
clean_consumed => true
tags => ["dlq"]
}
}
filter {
if "dlq" in [tags] {
json_encode {
source => "message_json"
target => "message"
remove_field => [ "message_json" ]
remove_tag => ["kubernetes", "docker", "kubernetes_filtered"]
}
mutate {
add_field => { "dlq_reason" => "%{[@metadata][dead_letter_queue][reason]}" }
}
}
}
output {
opensearch {
id => "kubernetes_es"
index => "%{[@metadata][index_field]}"
hosts => [ "endpoint" ]
manage_template => false
ssl => true
timeout => 200
retry_initial_interval => 100
retry_max_interval => 900
user => "${LS_ELASTICSEARCH_USER}"
password => "${LS_ELASTICSEARCH_PASSWORD}"
validate_after_inactivity => 60
}
}
vosmax
commented
1 year ago Having the same with 8.6.2 version
[WARN ] 2023-06-16 20:34:32.341 [dlq-flush-check] DeadLetterQueueWriter - unable to finalize segment
java.nio.file.NoSuchFileException: /usr/share/logstash/data/dead_letter_queue/app-logs-streams/5379.log.tmp -> /usr/share/logstash/data/dead_letter_queue/app-logs-streams/5379.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:416) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266) ~[?:?]
at java.nio.file.Files.move(Files.java:1432) ~[?:?]
at org.logstash.common.io.DeadLetterQueueWriter.finalizeSegment(DeadLetterQueueWriter.java:469) ~[logstash-core.jar:?]
at org.logstash.common.io.DeadLetterQueueWriter.flushCheck(DeadLetterQueueWriter.java:446) ~[logstash-core.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]$ ls -la /usr/share/logstash/data/dead_letter_queue/app-logs-streams/
total 128
drwxr-sr-x 2 logstash nogroup 58 Jun 16 17:45 .
drwxr-sr-x 7 logstash nogroup 112 Jun 15 12:55 ..
-rw-r--r-- 1 logstash nogroup 130973 Jun 16 17:45 5379.log
-rw-r--r-- 1 logstash nogroup 0 Jun 15 12:55 dlq_reader.lock
-rw-r--r-- 1 logstash nogroup 0 Jun 15 12:55 .lockIn another pod, I have the different situation
[WARN ] 2023-06-16 20:46:10.278 [dlq-flush-check] DeadLetterQueueWriter - unable to finalize segment
java.nio.file.NoSuchFileException: /usr/share/logstash/data/dead_letter_queue/app-logs-streams/3259.log.tmp -> /usr/share/logstash/data/dead_letter_queue/app-logs-streams/3259.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:416) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266) ~[?:?]
at java.nio.file.Files.move(Files.java:1432) ~[?:?]
at org.logstash.common.io.DeadLetterQueueWriter.finalizeSegment(DeadLetterQueueWriter.java:469) ~[logstash-core.jar:?]
at org.logstash.common.io.DeadLetterQueueWriter.flushCheck(DeadLetterQueueWriter.java:446) ~[logstash-core.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]$ ls -la /usr/share/logstash/data/dead_letter_queue/app-logs-streams/
total 0
drwxr-sr-x 2 logstash nogroup 42 Jun 16 10:15 .
drwxr-sr-x 7 logstash nogroup 112 Jun 15 12:55 ..
-rw-r--r-- 1 logstash nogroup 0 Jun 15 12:55 dlq_reader.lock
-rw-r--r-- 1 logstash nogroup 0 Jun 15 12:55 .lockHi @vosmax , is it possible to get your logstash config/arguments/env variables to see if there is any correlation?
vosmax
commented
1 year ago Hi @ukho-cfreeman, sure logstash.yml
log.level: info
http.host: "0.0.0.0"
xpack.monitoring.enabled: false
xpack.monitoring.collection.pipeline.details.enabled: false
dead_letter_queue.enable: true
dead_letter_queue.retain.age: 4h
path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"dlq.yml
input {
dead_letter_queue {
path => "/usr/share/logstash/data/dead_letter_queue"
commit_offsets => true
clean_consumed => true
pipeline_id => "app-logs-streams"
}
}
filter {
mutate {
add_field => {
"error_reason" => "%%{[@metadata][dead_letter_queue][reason]}"
"error_type" => "%%{[@metadata][dead_letter_queue][type]}"
}
}
prune {
whitelist_names => ["@timestamp","error","kubernetes","message"]
}
}
output {
elasticsearch {
hosts => [ <host>]
ssl => true
ssl_certificate_verification => false
user => "<user>"
password => "<password>"
index => "<index_name>"
manage_template => false
ilm_enabled => false
template_name => "<tpl_name>"
action => "create"
}
}In env vars only ES configuration
ukho-cfreeman
commented
1 year ago Thanks @vosmax , there doesn't seem to be any indication that this is based on our configuration. Would it be possible to see your dockerfile? Or if there is any correlation with mine:
FROM docker.elastic.co/logstash/logstash:8.7.1
RUN logstash-plugin install --version 3.0.3 logstash-filter-json_encode
RUN logstash-plugin install --version 2.2.1 logstash-input-kinesis
RUN logstash-plugin install --version 2.0.0 logstash-output-opensearch
USER root
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]
USER logstash
CMD ["/usr/local/bin/docker-entrypoint", "--config.reload.automatic", "-f", "/logstash/conf.d/config/logstash.conf"]
turbotankist
commented
1 year ago the same problem in 8.8.2 the support (the worst I've met) gave me the link for this issue and can't say anything.
Logstash information:
Please include the following information:
Plugins installed:
JVM (e.g.
java -version): not installed in docker imageOS version : Linux 9c7bb12feea2 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Steps to reproduce:
Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it. Environment variables:
I redacted filters etc, pl
Provide logs (if relevant): The following log lines get repeated several times a second