If we fail to delete the folder, we never attempt to delete again, raising a warning and moving on, leading to a proliferation of empty temp folders on systems that are prone to this issue (we believe that Windows boxes, in particular those running antivirus scans, are more prone to this issue than other systems).
The current cleanup implementation deliberately skips empty folders -
https://github.com/logstash-plugins/logstash-integration-aws/issues/28 describes a situation where the s3 output cannot delete a temporary folder, due to the folder being non-empty.
If we fail to delete the folder, we never attempt to delete again, raising a warning and moving on, leading to a proliferation of empty temp folders on systems that are prone to this issue (we believe that Windows boxes, in particular those running antivirus scans, are more prone to this issue than other systems).
The current cleanup implementation deliberately skips empty folders -
https://github.com/logstash-plugins/logstash-integration-aws/blob/main/lib/logstash/outputs/s3.rb#L399
and only cleans up folders containing partially completed upload files, and only if the
restoreoption is selected in the configuration.We should improve the cleanup process to also delete empty temporary folders.