logstash logs are cleared periodically

[fandrei]

It seems logstash overwrites it's log files on every restart, rather than appends to the file. Is it possible to fix this behavior?

[dpb587]

That shouldn't be happening. In what environment are you seeing this happen? How are you running the logstash and elasticsearch services?

[fandrei]

It's been running by the windows shipper service (installed from the package), all parameters are default except logs path. Win7 SP1 x64.

[fandrei]

It seems this behavior is controlled by winsw.exe It's necessary to change it's config and replace <log mode="reset"/> with <log mode="append"/>

[dpb587]

Ah, I was thinking the previously parsed logs in elasticsearch were getting deleted.

@mrdavidlaing, should we change the pre-configured log mode from reset to append? At some point it may cause issues due to filing up disk space because files are not rolling over and being archived. If no change is required, I think this issue should be closed.

[mrdavidlaing]

@dpb587, @fandrei - Lets change the winsw config to <log mode="roll-by-time"> - something like:

<log mode="roll-by-time">           <-- prevent reset
  <pattern>yyyyMMdd</pattern>  <-- roll daily
  <keepFiles>30</keepFiles>      <-- only keep 1 month's history
</log>

@fandrei - please could you check this syntax and submit a PR

[dpb587]

Oops. Didn't mean to close; thanks for re-opening.

[fandrei]

done. https://github.com/cityindex/logsearch-development-flow/pull/196

[fandrei]

@mrdavidlaing, should we also apply the same setting for the tunnel service?

[fandrei]

it still overwrites logs with the same date on every restart of logstash process; we need to use another log mode

[mrdavidlaing]

@fandrei -

1. Yes, when we find a log setting that works, then lets apply it everywhere
2. Bummer. Do any of the modes satisfy our "append" + roll + delete criteria? If not, perhaps its time to consider implementing #163

[fandrei]

@mrdavidlaing, roll-by-size seems to work correctly. Should I switch to it?

[dpb587]

@fandrei, I believe this has been fixed with #202 - can you confirm and close this issue; else I'll assume it fixed and close tomorrow.

[fandrei]

@dpb587, it seems the roll-by-time option doesn't work properly - I've already tried setting it manually

[mrdavidlaing]

@fandrei - The roll-by-time setting seems to work on the shippers I'm running, with the exception of the .wrapper.log file which doesn't roll.

Are you still seeing an issue here?

[fandrei]

@mrdavidlaing, on my test VM it still overwrites the ".err.log" on every restart

[sopel]

Closed as Incomplete due to project being retired to the CityIndex Attic.