search

logdna / logdna-agent

LogDNA Agent streams from log files to your LogDNA account. Works with Linux, Windows, and macOS Servers
https://logdna.com
MIT License
138 stars 55 forks source link

Deploying on Elastic beanstalk bug #212

Closed TheodoreCha closed 4 years ago

TheodoreCha commented 4 years ago

Hi,

Deploying app on EB, building fails with the following message:

Activity execution failed, because: Stopping logdna-agent............. Not stopped; may still be shutting down or shutdown may have failed Unable to stop, will not attempt to start (ElasticBeanstalk::ExternalInvocationError)

The weird thing is that first build fails but right after building fail, if I build the second with the same app, it works. so basically building two times in a role works.

Does anyone have an idea?

More info:

jalenplayvs commented 4 years ago

I can confirm that we have the same issue. Technically this should not be specific to Elastic Beanstalk, but it manifests itself very clearly as deployment failures when using Elastic Beanstalk. Possibly a bug in the process signal handling, as if it refuses to stop if it cannot flush the logs it is buffering (or something like that). Or maybe it just has a bug where Node has async code that is locked up and the process won't die.

Also, it would almost certainly be preferable to kill -9 here instead of failing, which would alleviate the deployment problem: https://github.com/logdna/logdna-agent/blob/master/tools/files/linux/init-script#L73-L75 The possibility that the agent is hung and not sending logs is still a concern, but I believe this behavior would be better regardless... and if this is simply a Node async deadlock with no other consequence, then likely kill -9 is the only reasonable solution.

darinspivey commented 4 years ago

Thanks for this report. Yes, we've received other reports of the shutdown not working on the first try (which may be a signal handling issue). We will look into it!

timbrownsf commented 4 years ago

Also having this issue, and re-deploying for a second time always fixes it as well.

[02_restart_logdna] : Activity execution failed, because: Stopping logdna-agent.............
  Not stopped; may still be shutting down or shutdown may have failed
  Unable to stop, will not attempt to start
   (ElasticBeanstalk::ExternalInvocationError)
darinspivey commented 4 years ago

We have tracked down the bug to be a hanging timer. For some reason, this only hangs when running the agent as a service, but that could be a difference between the binary (nexe build) of the agent versus not. Either way, the fix has been verified to work and a PR is incoming!

timbrownsf commented 4 years ago

Fantastic! This is causing large frustrations as every deploy (even to our staging environments) is failing so looking forward to the fix. Thank you.

darinspivey commented 4 years ago

We will cut a release for this by tomorrow. Thanks to all for contributing on this. Standby for the release...

smusali commented 4 years ago

LogDNA Agent v2.1.1 patch release containing the fix has already been published. Thanks all!

darinspivey commented 4 years ago

@timbrownsf and @CarlChaappletree feel free to touch base and let us know if this fix has indeed solved your problem as expected. Thanks!

jalenplayvs commented 4 years ago

I am still having this issue today with the latest release:

$ rpm -qa | grep logdna
logdna-agent-2.1.1-1.x86_64
TheodoreCha commented 4 years ago

@darinspivey

I don't see any errors I had before.

Thanks for the fixes!

smusali commented 4 years ago

@CarlChaappletree, @timbrownsf, @jalenplayvs's fix (#213) has been released in the newest version of the agent: LogDNA Agent v2.1.2

Thank y'all a lot!