logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

[Doc] tried to document better the setting #220

Closed andsel closed 4 years ago

andsel commented 4 years ago

Try to improve the description of time_file setting

andsel commented 4 years ago

Please @karenzone could give an eye if sounds more meaningful and readable, thanks

karenzone commented 4 years ago

@andsel Thanks for kicking this off.

At line 376:

If you define file_size you have a number of files in consideration of the section and the current tag.

Is this what it means?

If you define size_file, the section and the current tag also factor into the number of files created.

I know this isn't currently part of the PR, but it's closely related.

andsel commented 4 years ago

I try to explain the behaviour, probably you could come out with more meaningful description.

size_file rules when we have to split the file, to create various segments. This plugin accumulate data in a locally temporary file, and then when it reach the file_size it split, creating a segment and then it's able to push that segment ion the remote S3. Could be that for same tag (it's not clear to me what tag means, I imagine some common prefix) you have more split parts to upload.

time_file also rules when to split, but this time it consider the creation time. If the file was created 60 minutes, for example, split it.

The mixed behaviour time_and_size means: once one of the two rules match, then split it

karenzone commented 4 years ago

Thanks for the great explanation! I think you uncovered the piece I was missing.

Are these statements correct (not to go in docs "as is", but so I understand):

If you set rotation_strategy to size, it uses the value in size_file. If you set rotation_strategy to time, it uses the value in time_file. If you set rotation_strategy to size_and_time, it uses the values from size_file and time_file, and splits the file when either one matches.

If those statements are true, then that helps me know what I need to add to the docs for more clarity.

andsel commented 4 years ago

They are correct, exactly what the code do

elasticsearch-bot commented 4 years ago

Andrea Selva merged this into the following branches!

Branch Commits
master 8e2676933c4a43d8f78a62acf6352623dbdd6b22