Closed jeffmacdonald closed 6 years ago
If I'm understanding correctly, this looks like an issue with a rotated log file. Could you try out the latest beta, and see if it's fixed: v0.20-beta2
Hi, just acking that I've read this. I'm travelling but I look forward to giving this a try when I'm back home
I've deployed this and I'm still experiencing the same issues. I did some time co-relation
ps -o stime 4244
shows that the remote_syslog2 process started at 13:19However: this behaviour is not consistent. Sometimes it doesn't hang.
Doing a service remote_syslog stop; service remote_syslog start
doesn't appear to help either. Additionally after I do start/stop it still sits on FUTEX_WAIT
Actually, I now wonder if it's just getting hung up on some sort of open-file limitation. If you don't mind emailing into support@papertrailapp.com, we can get to the bottom of it a bit quicker. Then you can send over the full config, without it being public. (The output of lsof -p `pgrep remote_syslog`
, and/or adding --debug-log-cfg /tmp/remote_syslog.log
to the command-line, may tell us more.)
Roger that! I'll follow up tomorrow. Thanks
Solved issue by configuring stricter log file rotation, to prevent remote_syslog2 from bumping into inotify limits.
Hi.
I'm running remote_syslog2 on an AWS EB multi-container instance. The instance runs 2 copies of docker. Our apps log to stdout and files are saved to
/var/log/containers/ourapp-wsgi/*.log
and/var/log/containers/ourapp-celery-workers*.log
We find that after a few days, papertrail is no longer receiving our logs. Restarting remote_syslog2 on the app instances cause logs to start flowing again.
Here is some debugging I've tried to do.
So I dug a bit more with
ps -efL|grep remote_syslog|wc
and there were 132 results. Most looking something like thisroot 22924 1 2215 0 131 Sep21 ? 00:00:03 /usr/local/bin/remote_syslog -c /etc/log_files.yml --pid-file=/var/run/remote_syslog.pid --poll=true
Performing a
strace -f -p 2215
shows that the thread is "doing stuff" just fine.And so.. now I ask for your help.