The documentation for backup_add_prefix should mention that it does not have any effect unless the backup_to_bucket parameter is also specified.
In a related note it could be useful to mention that backup_to_bucket can be set to the same bucket that is being read from, and used in conjunction with backup_add_prefix to back up objects to a different path within the same bucket.
For instance a configuration such as the following could be used to store all objects in one bucket but avoid continually checking the older objects against the since DB to see if they had changed and/or avoid a new machine processing all old objects.
The documentation for backup_add_prefix should mention that it does not have any effect unless the backup_to_bucket parameter is also specified.
In a related note it could be useful to mention that backup_to_bucket can be set to the same bucket that is being read from, and used in conjunction with backup_add_prefix to back up objects to a different path within the same bucket.
For instance a configuration such as the following could be used to store all objects in one bucket but avoid continually checking the older objects against the since DB to see if they had changed and/or avoid a new machine processing all old objects.
input { s3 { bucket => "source-bucket" backup_to_bucket => "source-bucket" backup_add_prefix => "processed" delete => true exclude_pattern => "processed" } }
or alternately
input { s3 { source => "source-bucket" prefix => "data" backup_to_bucket => "source-bucket" backup_add_prefix => "processed" delete => true } }
Regards, Bill