Open niekosau opened 2 years ago
@niekosau can you show your logstash log of input.s3? May be i have a same problem
I'm using logstash 7.17.4 with plugin version 3.8.3
I think the comparison logic is the culprit here. In my case it is comparing timestamps which end with different local formats.
Before this I added
::File.open('/tmp/debug.log', 'a') { |file| file.write("object.last_modified: ", object.last_modified, ", log.last_modified: ", log.last_modified, "\n") }
Which gives me:
object.last_modified: 2021-06-25 16:38:23 +0000, log.last_modified: 2021-06-25 16:38:23 UTC
I'm not familiar with ruby but when comparing the dates it probably casts them to the above string representations which are not the same.
In the first step I fixed it on my side by comparing the unix time stamps
if object.last_modified.to_i == log.last_modified.to_i
Still this works not 100% as expected. The logic applied here is to save the latest timestamp seen in s3 to the sincedb. If you do not delete your files in the s3 you will always import all files which have the same timestamp as stored in the sincedb path. Which means you will import the latest file again and again.
To fix this issue I always add 1s to the timestamp written to the sincedb, after this line
since = since + 1
I'm not sure if there will be any side effects when simply adding +1 to the timestamp (at least it works for me now as expected).
One should probably fix the comparising logic.
Logstash information:
bin/logstash --version
)logstash 8.0.0
https://artifacts.elastic.co/packages/8.x/yum
systemd unit provided by package
shipped with logstash
OS version (
uname -a
if on a Unix-like system):Rocky linux 8.5 (4.18.0-348.7.1.el8_5.x86_64)
Description of the problem including expected versus actual behavior: files from bucket (radosgw on-premise) not removed, sincedb file not created/updatedSteps to reproduce: Configuration:
Tested older plugin versions, last working version:
3.5.0
Plugin downgraded by executing:bin/logstash-plugin install --version 3.5.0 logstash-input-s3
Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.