circus logging doesn't work alongside logrotate

[rednaks]

Hi,

I enabled logrotate for my circus services instead of circus logging rotate, the main reason was that i needed compression, a feature not available with circus.

but I noticed an issue that after the rotation happened circus is not logging anything. I made some tests and now I have a scenario to reproduce the issue.

When a log file is deleted, circus doesn't create a new one. if you touch to create manually the missing log file, circus will not take it into consideration and will not write in it. circusctl reload and circusctl restart don't help. the only solution is to restart circusd.

circus version: 0.17.1

I'll investigate more and if I have a PR i'll submit it :)

[rednaks]

looks like it's an issue with python.

import time
fp = open('/tmp/deleteme.log', 'a+')
while True:
    fp.write('hello\n')
    fp.flush()
    time.sleep(2)

If you delete the file while it's running, no more output. basically this is the code of the FileStream of circus. I donno what's the best approach.

1. test if the file exists ? -> logrotate will create a new one anyways
2. comparing inode maybe before writing ?

[biozz]

Hi @rednaks!

Thank you for reporting this, we definitely need a better documentation about logging. And I am not very good in this part of circus, but I would like to share my example, which might work for you.

I have circus configured like so:

stdout_stream.class = TimedRotatingFileStream
stdout_stream.filename = /var/log/foo/bar.log
stdout_stream.utc = True
stdout_stream.rotate_when = H
stdout_stream.rotate_interval = 1
stderr_stream.class = TimedRotatingFileStream
stderr_stream.filename = /var/log/foo/bar.log
stderr_stream.utc = True
stderr_stream.rotate_when = H
stderr_stream.rotate_interval = 1

There is also logrotate inside the container which is configured as:

/var/log/foo/*.log {
    rotate 1
    size 1G
    daily
    missingok
    notifempty
}

And it is working as expected (almost): I have current log file bar.log and a ton of daily bar-%Y%m%d.gz files. I added "almost", because I don't know what circus does in this case.

So I assume there might be some permissions/configuration issues in your setup, could you try to check it again?

[johanfforsberg]

Maybe the "copytruncate" setting in logrotate would help? I believe it will copy the files instead of renaming them, and then truncate the original logfile. This should allow python to keep writing to the file, as if nothing happened.