Process hung with socket opened in LogDNA 2.1.8-beta

[AeroNotix]

Where is the code/issue tracker for the latest version? Should we make tickets on here.

I just found a logdna-agent:2.1.8-beta process seemingly hung, socket open to LogDNA's IP in ESTAB, but with no data flowing over the socket. Once I restarted the container, logs started flowing.

[CJP10]

Do you have the agent logs for this period of time it was halted.

[AeroNotix]

The agent container had no logs at all, as it happened.

[CJP10]

Do you happen to know what the last few log lines were in the container?

[AeroNotix]

The container itself had no log output - we installed using the helm chart. I tried adding various flags to the executable to get it to start logging but nothing seemed to output anything on stdout. Is there a flag I could add to make it output logs?

Will source code be available eventually?

[AeroNotix]

Unfortunately, we've had to revert back to using the latest tag as many logdna-agent:2.1.8-beta containers were ending up in this hung state.

Where is the code? Will the new version be open-sourced?

[andkon]

@AeroNotix thanks for the updates. I'm sorry that it's been a persistent issue – can you point us to the Helm chart you've been using? Any other info about your environment would be helpful – I'm at andrew.konoff@logdna.com if you'd rather share it by email.

We're planning on open sourcing the v2 repo in time for general availability; in the meantime, we're still working on docs.

[CJP10]

@AeroNotix the only supported installation method for using the beta agent is the following yaml https://github.com/logdna/logdna-agent/blob/master/logdna-agent-v2-beta.yaml.

Give that a shot and see if your problem persists.

[AeroNotix]

So the way we deployed this was keeping the previous logdna helm chart and updating the tag for which docker image to use. The new chart doesn't seem like it would have an impact, aside from the RUST_LOG environment variable.

[andkon]

@AeroNotix There are a number of changes to how v2 works that likely won't mesh well with the Helm chart that's out there, including the API version, and features that rely on RBAC. If you could try to install the agent with the following docs, it would help us understand how to solve your issue, or if it's just a result of using the Helm chart: https://docs.logdna.com/docs/logdna-agent-kubernetes

[AeroNotix]

When we get a window to push this out again, I will. Why would it push for a period of time - then stop? One would think if it needed a specific API version or RBAC features it wouldn't start.

[andkon]

Thank you – and it’s a great question. I’m not sure! But at very least, we could rule out these configuration/helm issues and discover what the underlying cause is from there.

Btw, what version of Kubernetes are you on?

On Feb 12, 2020, at 5:30 PM, Aaron France notifications@github.com wrote:

When we get a window to push this out again, I will. Why would it push for a period of time - then stop? One would think if it needed a specific API version or RBAC features it wouldn't start.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/logdna/logdna-agent/issues/155?email_source=notifications&email_token=AAPYJZ6VL3CUV2F53DNI26TRCR2ADA5CNFSM4KSYK4T2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELSUMBA#issuecomment-585451012, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPYJZZEI656AD5VG7W53ZLRCR2ADANCNFSM4KSYK4TQ.

[AeroNotix]

v1.13.10-gke.0

[jakedipity]

@AeroNotix We've since updated the agent to 2.1.9 and have updated our helm chart to be compatible with the latest 2.x.x agent. Have you had a chance to update and have you seen any of the same issues?

Thanks!

[sgwilson2674]

Closing the issue due to inactivity. If additional questions remain, please open another issue and we’ll be sure to respond. If additional kubernetes questions, please visit this repo which is used for our k8s agent: https://github.com/logdna/logdna-agent-v2