Tracks every created temp files make sure they are removed.
Introduced temp_files container (FileRepository locks for thread safety) to keep and track all rotated files. When file is removed from track list?
when recently created file becomes stale, means no event written to the file
When file and tempdir are physically deleted?
when uploaded to S3 - both temp dir and file are deleted.
when becomes stale: no event written to the file during 15 mins - both temp dir and file are deleted.
!NEW: deleting file (file.delete!) didn't succeed, means third party process (especially on Windows, security scanners) is keeping open the IO and Logstash removes the file. Physically, for this case, file will be deleted but temp dir stays. - both temp dir is deleted.
asdas
+def remove_staled_files
+ with_lock do |factory|
+ factory.temp_files = factory.temp_files.delete_if do |temp_file|
+ is_staled = temp_file.size == 0 && (Time.now - temp_file.ctime > @stale_time)
+ is_temp_dir_empty = false
+ begin
+ # checking Dir emptiness and remove file
+ # reading file and checking size doesn't make sense as it will not possible after temp_file.size == 0 filter
+ temp_file.delete! if is_staled
+ is_temp_dir_empty = Dir.empty?(::File.join(temp_file.temp_path, temp_file.prefix)) unless is_staled
+ temp_file.delete! if is_temp_dir_empty
+ rescue => e
+ @logger.error("An error occurred while sweeping temp dir.", :exception => e.class, :message => e.message, :path => temp_file.path, :backtrace => e.backtrace)
+ end
+ is_staled || is_temp_dir_empty
+ end
+ # mark as deleted once we finish tracking all temp files created
+ @is_deleted = factory.temp_files.length == 0
+ end
+end
Moves the periodic stale sweeper to higher level, same level as periodic file rotator to make KISS. Also, confusing codes are cleaned such as properly namings (ex: periodic file rotator, periodic file sweeper), correcting misleading comments/log-descriptions.
Description
This PR mainly focuses on two changes:
temp_files
container (FileRepository
locks for thread safety) to keep and track all rotated files. When file is removed from track list?When file and tempdir are physically deleted?
file.delete!
) didn't succeed, means third party process (especially on Windows, security scanners) is keeping open the IO and Logstash removes the file. Physically, for this case, file will be deleted but temp dir stays. - both temp dir is deleted.asdas