The current sincedb implementation of this plugin only relies on the object key and doesn't use the file offset at all, so when we stop logstash in a middle of reading a file we don't have the choice to read the file back at the beginning causing duplicates in the log stream.
We should investigate if we could use the filewatch plugins to actually do the file reading and the S3 input could act as a downloading agent.
The current sincedb implementation of this plugin only relies on the object key and doesn't use the file offset at all, so when we stop logstash in a middle of reading a file we don't have the choice to read the file back at the beginning causing duplicates in the log stream.
We should investigate if we could use the filewatch plugins to actually do the file reading and the S3 input could act as a downloading agent.