search

logdna / logdna-agent-v2

The blazingly fast, resource efficient log collection client
https://logdna.com
MIT License
66 stars 46 forks source link

application logs #52

Closed danikostyal closed 3 years ago

danikostyal commented 3 years ago

Hi! We are trying to use the IBM Log Analysis with LogDNA to gather logs from some MobileFirst pods running in the IBM Kubernetes Service. We are having trouble get LogDNA service to read the application logs which are held in /opt/ibm/wlp/usr/servers/mfp/logs/ path. We added the LOGDNA_LOG_DIRS variable but still no luck. The MobileFirst app sends logs only to file, not to STDOUT. Is this the problem? Any help is much appreciated.

jakedipity commented 3 years ago

For Kubernetes we recommend using our rust developed agent logdna-agent-v2. The naming scheme is a little messy at the moment and we're hoping to clean this up in the coming months.

Yes that setup is problematic for both agents. Normally the pods will write their logs to STDOUT and Kubernetes / the container runtime environment will collect the logs and store them on the host node. The agent runs in its own isolated nodes and mounts these logs from the host node. If you are writing directly to a file the logs are most likely only being stored in your MobileFirst pods as opposed to the host node.

I can think of a few creative ways to get around this but to get the best experience with Kubernetes and other tools for Kubernetes, I would recommend writing your logs to STDOUT.

danikostyal commented 3 years ago

Thank you for the quick reply, much appreciate it. Yes, we are using logdna-agent-v2. We are not able at the moment to change how the app writes the logs. As a workaround, I was thinking of streaming to a sidecar container. Any thoughts on this, would it work? Do you have any other options we could try? Many thanks!

jakedipity commented 3 years ago

That would probably be the cleanest workaround for your case. It should work since you're still relying on Kubernetes to handle log management which means the logs will end up in the correct place for the agent to ingest. The only thing that might be different is the container metadata we collect from the logs; the logs will have the metadata for the sidecar container as opposed to the main container.

The documentation there seems to provide a good starting place for this approach. I'm going to close this issue, but let us know if it works or not. If not we can think of something else!