nickdnk
commented
1 year ago I'm getting a lot of this as well. Again only on the worker environment and always only one of the two.
Last 100 of the problematic instance. There are no errors in any of the other log files I can find.
eb-engine.log
2023/08/16 14:57:51.260683 [INFO] Running command /bin/sh -c systemctl stop xray.service
2023/08/16 14:57:51.274434 [INFO] Executing instruction: stop proxy
2023/08/16 14:57:51.274482 [INFO] Running command /bin/sh -c systemctl show -p PartOf httpd.service
2023/08/16 14:57:51.287472 [WARN] deregisterProcess Warning: process httpd is not registered, skipping...
2023/08/16 14:57:51.287505 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2023/08/16 14:57:51.300385 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2023/08/16 14:57:51.309818 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2023/08/16 14:57:51.320137 [INFO] Running command /bin/sh -c systemctl stop nginx.service
2023/08/16 14:57:51.467114 [INFO] Running command /bin/sh -c systemctl disable nginx.service
2023/08/16 14:57:51.704484 [INFO] Running command /bin/sh -c systemctl daemon-reload
2023/08/16 14:57:51.944630 [INFO] Running command /bin/sh -c systemctl reset-failed
2023/08/16 14:57:51.953801 [INFO] Executing instruction: stop php-fpm
2023/08/16 14:57:51.953835 [INFO] Running command /bin/sh -c systemctl show -p PartOf php-fpm.service
2023/08/16 14:57:51.966522 [INFO] Running command /bin/sh -c systemctl stop php-fpm.service
2023/08/16 14:57:52.057020 [INFO] Executing instruction: FlipApplication
2023/08/16 14:57:52.057047 [INFO] Removing /var/app/current/ if it exists
2023/08/16 14:57:52.363265 [INFO] Renaming /var/app/staging/ to /var/app/current/
2023/08/16 14:57:52.363526 [INFO] create soft link from /var/app/current/ to /var/www/html
2023/08/16 14:57:52.363573 [INFO] Executing instruction: start X-Ray
2023/08/16 14:57:52.363579 [INFO] X-Ray is not enabled.
2023/08/16 14:57:52.363582 [INFO] Executing instruction: start php-fpm
2023/08/16 14:57:52.363779 [INFO] Running command /bin/sh -c systemctl show -p PartOf php-fpm.service
2023/08/16 14:57:52.379370 [WARN] Warning: process php-fpm is already registered...
Deregistering the process ...
2023/08/16 14:57:52.379402 [INFO] Running command /bin/sh -c systemctl show -p PartOf php-fpm.service
2023/08/16 14:57:52.393227 [INFO] Running command /bin/sh -c systemctl is-active php-fpm.service
2023/08/16 14:57:52.404069 [INFO] Running command /bin/sh -c systemctl disable php-fpm.service
2023/08/16 14:57:52.614543 [INFO] Running command /bin/sh -c systemctl daemon-reload
2023/08/16 14:57:52.837774 [INFO] Running command /bin/sh -c systemctl reset-failed
2023/08/16 14:57:52.849583 [INFO] Running command /bin/sh -c systemctl daemon-reload
2023/08/16 14:57:53.050823 [INFO] Running command /bin/sh -c systemctl reset-failed
2023/08/16 14:57:53.059984 [INFO] Running command /bin/sh -c systemctl show -p PartOf php-fpm.service
2023/08/16 14:57:53.074126 [INFO] Running command /bin/sh -c systemctl is-active php-fpm.service
2023/08/16 14:57:53.083895 [INFO] Running command /bin/sh -c systemctl start php-fpm.service
2023/08/16 14:57:53.267227 [INFO] Executing instruction: start proxy with new configuration
2023/08/16 14:57:53.267276 [INFO] Running command /bin/sh -c /usr/sbin/nginx -t -c /var/proxy/staging/nginx/nginx.conf
2023/08/16 14:57:53.291203 [INFO] nginx: the configuration file /var/proxy/staging/nginx/nginx.conf syntax is ok
nginx: configuration file /var/proxy/staging/nginx/nginx.conf test is successful
2023/08/16 14:57:53.291368 [INFO] Running command /bin/sh -c cp -rp /var/proxy/staging/nginx/* /etc/nginx
2023/08/16 14:57:53.294811 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2023/08/16 14:57:53.311070 [INFO] Running command /bin/sh -c systemctl daemon-reload
2023/08/16 14:57:53.489838 [INFO] Running command /bin/sh -c systemctl reset-failed
2023/08/16 14:57:53.499430 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2023/08/16 14:57:53.513675 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2023/08/16 14:57:53.523517 [INFO] Running command /bin/sh -c systemctl start nginx.service
2023/08/16 14:57:53.647258 [INFO] Executing instruction: configureSqsd
2023/08/16 14:57:53.650422 [INFO] get sqsd conf from cfn metadata and write into sqsd conf file ...
2023/08/16 14:57:53.651915 [INFO] Executing instruction: startSqsd
2023/08/16 14:57:53.651935 [INFO] Running command /bin/sh -c systemctl show -p PartOf sqsd.service
2023/08/16 14:57:53.663928 [INFO] Running command /bin/sh -c systemctl is-active sqsd.service
2023/08/16 14:57:53.672727 [INFO] Running command /bin/sh -c systemctl start sqsd.service
2023/08/16 14:57:59.859633 [INFO] Executing instruction: Track pids in healthd
2023/08/16 14:57:59.859649 [INFO] This is an enhanced health env...
2023/08/16 14:57:59.859661 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
2023/08/16 14:57:59.870997 [INFO] nginx.service cfn-hup.service amazon-cloudwatch-agent.service php-fpm.service sqsd.service healthd.service
2023/08/16 14:57:59.871029 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2023/08/16 14:57:59.881275 [INFO]
2023/08/16 14:57:59.881499 [INFO] Executing instruction: RunAppDeployPostDeployHooks
2023/08/16 14:57:59.881585 [INFO] Executing platform hooks in .platform/hooks/postdeploy/
2023/08/16 14:57:59.881612 [INFO] The dir .platform/hooks/postdeploy/ does not exist
2023/08/16 14:57:59.881615 [INFO] Finished running scripts in /var/app/current/.platform/hooks/postdeploy
2023/08/16 14:57:59.881619 [INFO] Executing cleanup logic
2023/08/16 14:57:59.881705 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment: You included a 'vendor' folder in your source bundle. The deployment ignored 'composer.json' and didn't install Composer dependencies.","timestamp":1692197866757,"severity":"INFO"},{"msg":"Instance deployment completed successfully.","timestamp":1692197879881,"severity":"INFO"}]}]}
2023/08/16 14:57:59.881862 [INFO] Platform Engine finished execution on command: app-deploy
2023/08/16 14:58:05.105310 [INFO] Starting...
2023/08/16 14:58:05.105381 [INFO] Starting EBPlatform-PlatformEngine
2023/08/16 14:58:05.105401 [INFO] reading event message file
2023/08/16 14:58:05.105703 [INFO] Engine received EB command cfn-hup-exec
2023/08/16 14:58:05.174719 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-central-1:REDACTED:stack/awseb-e-REDACTED/REDACTED -r AWSEBAutoScalingGroup --region eu-central-1
2023/08/16 14:58:05.508542 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-central-1:REDACTED:stack/awseb-e-REDACTED/REDACTED -r AWSEBBeanstalkMetadata --region eu-central-1
2023/08/16 14:58:05.837325 [INFO] checking whether command app-deploy is applicable to this instance...
2023/08/16 14:58:05.837343 [INFO] this command is not applicable to the instance, thus instance shouldn't execute command
2023/08/16 14:58:05.837347 [INFO] skip command app-deploy for this instance...
2023/08/16 14:58:05.837367 [ERROR] Ignoring not applicable command.
2023/08/16 14:58:05.837370 [INFO] Executing cleanup logic
2023/08/16 14:58:05.837468 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Ignoring not applicable command.","returncode":0,"events":[]}]}
2023/08/16 14:58:05.837673 [INFO] Platform Engine finished execution on command: app-deploy
2023/08/16 15:08:34.115957 [INFO] Starting...
2023/08/16 15:08:34.116018 [INFO] Starting EBPlatform-PlatformEngine
2023/08/16 15:08:34.116038 [INFO] reading event message file
2023/08/16 15:08:34.116228 [INFO] Engine received EB command cfn-hup-exec
2023/08/16 15:08:34.204704 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-central-1:REDACTED:stack/awseb-e-REDACTED/REDACTED -r AWSEBAutoScalingGroup --region eu-central-1
2023/08/16 15:08:34.543080 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:eu-central-1:REDACTED:stack/awseb-e-REDACTED/REDACTED -r AWSEBBeanstalkMetadata --region eu-central-1
2023/08/16 15:08:34.889230 [INFO] checking whether command tail-log is applicable to this instance...
2023/08/16 15:08:34.889243 [INFO] this command is applicable to the instance, thus instance should execute command
2023/08/16 15:08:34.889246 [INFO] Engine command: (tail-log)
2023/08/16 15:08:34.889291 [INFO] Executing instruction: GetTailLogs
2023/08/16 15:08:34.889294 [INFO] Tail Logs...
2023/08/16 15:08:34.889836 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-engine.log
RachelZhao
PeterCai7
tehmaestro
morgano86
Hello
After upgrading our platform to PHP 8.2 (4.0.0), we consistently have issues with our worker environment updating only one instance and letting the other one hang indefinitely. The pipeline considers the deployment successful, but health shows this:
Terminating the instance will launch a new one which works, which points to some kind of deployment command stalling. Rebooting the instance also works.
Relevant log from the hanging instance:
eb-engine.log