zodern / mup-aws-beanstalk

Deploy Meteor apps to AWS Elastic Beanstalk using Meteor Up
127 stars 56 forks source link

Problem deploying with graceful shutdown #171

Open tcastelli opened 2 years ago

tcastelli commented 2 years ago

We are currently trying to deploy a basic app that needs to listen to SIGTERM before closing. The configuration file has this options (Meteor 2.5.8).

 {
              {
                namespace: 'aws:elasticbeanstalk:cloudwatch:logs',
                option: 'StreamLogs',
                value: 'true'
            },
            {
                namespace: 'aws:elasticbeanstalk:cloudwatch:logs',
                option: 'DeleteOnTerminate',
                value: 'true'
            },
            {
                namespace: 'aws:elasticbeanstalk:cloudwatch:logs:health',
                option: 'HealthStreamingEnabled',
                value: 'true'
            },
            {
                namespace: 'aws:elasticbeanstalk:cloudwatch:logs:health',
                option: 'DeleteOnTerminate',
                value: 'true'
            },
            {
                namespace: 'aws:elasticbeanstalk:environment',
                option: 'LoadBalancerIsShared',
                value: 'true'
            },
            {
                namespace: 'aws:elbv2:loadbalancer',
                option: 'SharedLoadBalancer',
                value: 'xxxx'
            },
            {
                namespace: 'aws:elasticbeanstalk:command',
                option: 'DeploymentPolicy',
                value: 'AllAtOnce'
            },

           {
               namespace: 'aws:autoscaling:updatepolicy:rollingupdate',
               option: 'RollingUpdateType',
               value: 'Immutable'
           },

The first thing we have noticed is that the deployment of a new version sometimes doesn't work, because it leaves the instance with Severe status and HealthCHeck never resolve (we see a socket hang up response in the logs), SIGTERM doesn't seem to be received, since our logging doesn't say anything. Checking the eb-hooks log says the .platform/hooks/prebuild/45node.sh is the last thing trying to do but no errors after that.

Weirdly, when the deployment works, we can see in the logs that the SIGTERM is received two times at the same time(printed two times), but everything seems to finish correctly.

ANy ideas of what could be happening? @zodern

tcastelli commented 2 years ago

I got some useful info from zodern, saying that AllAtOnce type might not be supported. I since then switched to Immutable and the deployments are finish properly, but can't see any info in the logs about the SIGTERM being received. At this point I'm not sure if it's processed but not logged or not received at all. Will update later

tcastelli commented 2 years ago

So, after doing serveral tests, it seems that console.logs or similar used on the process.on("SIGTERM", xxxx will not be stored in cloudwatch, but the rest of the code is being executed and working.

Also, not sure why but we still receive two SIGTERMS whenever the instance is replaced