File Output Enhancement - file rotation

[red8383light]

Have seen several questions about this but we do require logstash to have the ability to output to file and limit file size. Example: Only write files to size 250mb or 250,000 lines.

Using timebased file rotation has resulted in Gb size files every hour, and the timestamp will not rotate on the 10 minute (YYYYMMddHHm) so it rotates every minute which generates more files than we would like. In some of our scenarios, 10 minutes would still generate files that are too large.

We also would like the ability to place a rotate counter of some kind within the file name rather than append to the end of the file. Example: /var/log/mylogfile.%{file_counter}.%{host}.%{YYYYMMddHHm}.json

Our current requirement is that the file must end in ".json".

[jsvd]

this is essentially file rotation, which is already performed by the decades old, tried and tested tools like logrotate. I'm not saying this plugin will never have this kind of feature but it's definitely a low priority and I don't see major scenarios of rotation that logrotate doesn't take care of.

[timo-klarshift]

this is essentially file rotation, which is already performed by the decades old, tried and tested tools like logrotate. I'm not saying this plugin will never have this kind of feature but it's definitely a low priority and I don't see major scenarios of rotation that logrotate doesn't take care of.

The problem with that is, that logstash has to be restarted because it does not pick up the changes done by logrotate.

[sarkortrantor]

@timo-klarshift

The problem with that is, that logstash has to be restarted because it does not pick up the changes done by logrotate.

regarding this observation this might be the same issue described in https://github.com/logstash-plugins/logstash-output-file/issues/81 and for which I submitted a PR a while back. you might want to patch your plugin.

[XavM]

this is essentially file rotation, which is already performed by the decades old, tried and tested tools like logrotate. I'm not saying this plugin will never have this kind of feature but it's definitely a low priority and I don't see major scenarios of rotation that logrotate doesn't take care of.

The problem with that is, that logstash has to be restarted because it does not pick up the changes done by logrotate.

I could be wrong but I think this is an exact use case for logrotate copytruncate :

Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.