search

logdna / logdna-agent

LogDNA Agent streams from log files to your LogDNA account. Works with Linux, Windows, and macOS Servers
https://logdna.com
MIT License
137 stars 55 forks source link

Support for inexistent files #13

Closed iagomelanias closed 7 years ago

iagomelanias commented 7 years ago

If we specify a file that does not exist, the logdna-agent will not keep running waiting the file to be created. That is really common when working with deploys.

We have a environment using a deploy tool. Every time a new deploy is made, a new log file is created inside the project, and the old one is moved. What happens is if we make a deploy, the logs will not be sent anymore, because the file was moved the and logdna-agent will not find it.

So we need to create a empty log file by default in the VCS repository. What is not a recommended practice.

Example: Default environment: The logs are in /deploys/current/var/logfile.log.

A new deploy is made: The file /deploys/current/var/logfile.log is moved to another folder. And the logdna-agent stops working.

And the new logfile.log is created only when any information is logged.

So, would be great if the agent keeps running even if the file does not exists. This kind of situation is really common.

leeliu commented 7 years ago

Interesting. We do support logdir which should monitor an entire folder for new files (rescans every minute), instead of monitoring a single file. Did you try adding logdna-agent -d /deploys/current/var/ and see if that'll work?

The only scenarios where this wouldn't work, off the top of my head, is if: 1) /deploys/current/var/ doesn't exist either at logdna-agent startup 2) logfile.log is moved but not immediately replaced by a new file of the same name

remigabillet commented 7 years ago

I have a similar issue where I'm tailing specific log files where the directory they're located in have way too many noisy logs that I don't want to forward to logdna. The problem is that most log files are created when needed so the agent doesnt catch them and I have to manually restart agents. It would be really nice if the agent could pick up specific files every minute just like it does with directories.

iagomelanias commented 7 years ago

@leeliu The problem of using this approach is because we have many logs in /deploys/current/var/, like prod.log, console.log, etc. And we only wanna send the relevant log files, that is basically the prod.log file.

And about the file, it's removed and only replaced when new log lines are inserted. And as the file does not exist when a new deploy is made, the file needs to be created. So there is a delay between the old file is moved and a new one is created. And as i said, that is a really common environment when working with application deployment.

Using Chef for deployments, for example, we have a releases folder containing the last 5 deploys, and the folder current, that links to the specific version of the release that is inside the releases folder. We would need to create a empty file every time a new deploy is made. And that is not something good.

I think the agent should identify when this happens, and do not simply send a exception saying the file does not exist and stops everything, what actually happens today. A simple "ok, lets skip this file by now and lets try it again later" would be awesome. Thanks :)

And that is what happens with us too @regabi. We don't wanna send every log file that is in the folder to the LogDNA, only some of them. And as they're created when needed, the agent simply does not pick them after a deployment.

leeliu commented 7 years ago

Got it. Give me some time to make the necessary changes.

iagomelanias commented 7 years ago

Thank you @leeliu. Loving to migrate from Logentries to the LogDNA.

leeliu commented 7 years ago

@iagomelanias Okay, this commit will now continue to scan for files every minute, even if it didn't exist at startup. I also confirmed after some testing that if you move/rename/delete a file that is being tailed, it will keep tailing even if a new file with the same name isn't immediately created. As far as I can tell, it'll wait indefinitely for the new file to show up.

remigabillet commented 7 years ago

Thank you @leeliu. I can't believe how fast you guys respond to requests, it's impressive. When can I expect the apt repo to be updated with this fix?

iagomelanias commented 7 years ago

Thank you again @leeliu. Awesome support. Updating our agent version.

@regabi It was already released. I've updated our agent version by running the commands above:

apt-get update
apt-get install --only-upgrade logdna-agent

👍

leeliu commented 7 years ago

Thanks @iagomelanias for helping with the apt commands. I should've indicated next steps in my last msg.

Anyway, the release is live on the repos but we haven't sent out auto update notifications to connected agents yet. We typically delay this for a few days in case of a bad release. But if you wait for it, all your agents will eventually update themselves to the latest version.

remigabillet commented 7 years ago

Oh, I didn't realize you had automatic updates, it's very cool.

Out of curiosity, how do I know when the agent gets updated, is the version number going to be incremented? (I'm on 1.3.4 right now).

On Thu, Sep 22, 2016 at 11:45 AM, Lee Liu notifications@github.com wrote:

Thanks @iagomelanias https://github.com/iagomelanias for helping with the apt commands. I should've indicated next steps in my last msg.

Anyway, the release is live on the repos but we haven't sent out auto update notifications to connected agents yet. We typically delay this for a few days in case of a bad release. But if you wait an it, all your agents will update themselves to the latest version.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/logdna/logdna-agent/issues/13#issuecomment-248992680, or mute the thread https://github.com/notifications/unsubscribe-auth/AABPT_QIOMPdDej7ul4k9rfrc-BWS9xcks5qsszngaJpZM4KBauO .

iagomelanias commented 7 years ago

@leeliu me too.

And just reporting a bug, the current version shown in the command is 1.3.6 while it was relased as 1.3.5 in the releases tab.

leeliu commented 7 years ago

@regabi Yes, the agent version will be 1.3.6 when autoupdate finishes. You can check by logdna-agent -v

@iagomelanias Oops...thanks for catching that! We skipped 1.3.5, it was a quick fix for a customer and subsequently never publicly released it. I've retagged it as 1.3.6.

iagomelanias commented 7 years ago

@leeliu Thanks again! 😄

leeliu commented 7 years ago

Regarding autoupdate, I've always hated having to manage versions of agents (mainly coming from New Relic). The autoupdate mechanism just uses yum/apt to do the actual update which I thought was a quick win and fairly easy to implement. And you can always turn it off autoupdate=0 or remove the yum/apt repo if you need the control over updates.