search

logdna / logdna-agent

LogDNA Agent streams from log files to your LogDNA account. Works with Linux, Windows, and macOS Servers
https://logdna.com
MIT License
137 stars 55 forks source link

When agent reconnects, it doesn't seem to pick logs again #101

Closed scalp42 closed 5 years ago

scalp42 commented 5 years ago

Working through the trial, we just noticed that the agent does not pick up new lines added to log files after it reconnects:

Logs from the agent:

[190612 01:15:22] Connected to logs.logdna.com:443 (13.82.132.160) (SSL)
[190612 01:15:22] Streaming /var/log/apt/history.log: 1 file(s)
[190612 01:15:22] Streaming /var/log/logdna-agent.log: 1 file(s)
[190612 02:01:16] Disconnected from server: 1006:
[190612 02:01:17] Attempting to connect #1 to logs.logdna.com:443 (SSL) using xxxx:xxxx...
[190612 02:01:18] Connected to logs.logdna.com:443 (52.170.85.196) (SSL)
[190612 02:01:18] Streaming resumed: 2 file(s)

Logs from /var/log/apt/history.log:

root@default-ubuntu-1804:/var/log# tail -f /var/log/apt/history.log
Commandline: apt install ncdu
Requested-By: vagrant (1000)
Install: ncdu:amd64 (1.12-1)
End-Date: 2019-06-12  01:16:59

Start-Date: 2019-06-12  02:03:30
Commandline: apt install --yes iotop
Requested-By: vagrant (1000)
Install: iotop:amd64 (0.6-2)
End-Date: 2019-06-12  02:03:31

You can see a new package was installed at 02:03:31, yet nothing is happening even though agent reconnected at 02:01:18.

Confirmed through UI as well:

Screen Shot 2019-06-11 at 19 05 30

I can connect fine to the endpoint from the box:

root@default-ubuntu-1804:/var/log# date; nc -w 1 -v -z logs.logdna.com 443
Wed Jun 12 02:11:46 UTC 2019
Connection to logs.logdna.com 443 port [tcp/https] succeeded!

Any ideas?

scalp42 commented 5 years ago

Looks like it finally appeared Wed Jun 12 02:26:41 UTC 2019.

dchai76 commented 5 years ago

Anthony, can I ask how you were checking for the log line to appear? Were you just leaving the UI window open?

scalp42 commented 5 years ago

Yes, tailing logs in the UI.

dchai76 commented 5 years ago

Unfortunately, we currently have an issue with reliability for live tailing logs that should be resolved soon. Your logs are being picked up, but you may need to refresh the UI page to be able to see them. There is usually a slight delay (on the order of minutes) before they appear, but certainly not on the order of hours (if that is what you saw). We're tracking the live tail issue elsewhere, so let me close this issue, but if you do see a long delay again, or if you do see logs not being picked up at all, please let us know.