search

Azure / logstash-output-kusto

Logstash output for Kusto
Apache License 2.0
13 stars 14 forks source link

Fix duplicated ingestion when failed to delete the tmp file #33

Closed unbuilt closed 3 years ago

unbuilt commented 3 years ago

When ingestion succeeded but File.Delete failed, the upload will retry, and duplicated items will be ingested. To avoid this duplicated ingestion problem, separating the ingesting and deleting in retry.

Example: [2020-12-21T03:03:34,036][INFO ][logstash.outputs.kusto ][main][033d4988a1f16e14e226afbe4863982dbeb6db44bc96720d6a3dabc27342edc0] Closing file C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog [2020-12-21T03:03:34,615][INFO ][logstash.outputs.kusto ][main][033d4988a1f16e14e226afbe4863982dbeb6db44bc96720d6a3dabc27342edc0] Opening file {:path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog"} [2020-12-21T03:03:34,630][ERROR][logstash.outputs.kusto ][main] Uploading failed, retrying. {:exception=>Errno::EACCES, :message=>"Permission denied - C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :backtrace=>["org/jruby/RubyFile.java:1293:in delete'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:101:in upload'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:73:in block in upload_async'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.7/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]} [2020-12-21T03:03:48,607][ERROR][logstash.outputs.kusto ][main] Uploading failed, retrying. {:exception=>Errno::EACCES, :message=>"Permission denied - C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :backtrace=>["org/jruby/RubyFile.java:1293:in delete'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:101:in upload'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:73:in block in upload_async'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.7/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]} [2020-12-21T03:03:55,615][ERROR][logstash.outputs.kusto ][main] Uploading failed, retrying. {:exception=>Errno::EACCES, :message=>"Permission denied - C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :backtrace=>["org/jruby/RubyFile.java:1293:in delete'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:101:in upload'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:73:in block in upload_async'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.7/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]} [2020-12-21T03:03:56,243][INFO ][logstash.outputs.kusto ][main][033d4988a1f16e14e226afbe4863982dbeb6db44bc96720d6a3dabc27342edc0] Closing file C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog [2020-12-21T03:03:59,057][ERROR][logstash.outputs.kusto ][main] File doesn't exist! Unrecoverable error. {:exception=>Errno::ENOENT, :message=>"No such file or directory - No such file or directory - C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-02.txt.gcsdmlog.DMLog", :backtrace=>["org/jruby/RubyFileTest.java:247:in size'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:81:in upload'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-kusto-0.4.0/lib/logstash/outputs/kusto/ingestor.rb:73:in block in upload_async'", "C:/LogPipeline/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.7/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]} [2020-12-21T03:04:02,664][INFO ][logstash.outputs.kusto ][main][033d4988a1f16e14e226afbe4863982dbeb6db44bc96720d6a3dabc27342edc0] Opening file {:path=>"C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-04.txt.gcsdmlog.DMLog"} [2020-12-21T03:04:51,386][INFO ][logstash.outputs.kusto ][main][033d4988a1f16e14e226afbe4863982dbeb6db44bc96720d6a3dabc27342edc0] Closing file C:/LogPipeline/logstash-7.10.1/tmp/kusto/2020-12-21-03-03.txt.gcsdmlog.DMLog

unbuilt commented 3 years ago

Looks good! But it should have some tests verifying the new behavior. See the spec folder

Totally agree. Thanks for your comments. However, it doesn't look straightforward, need much time to figure it out. So close it now.