aws / elastic-beanstalk-roadmap

AWS Elastic Beanstalk roadmap
https://aws.amazon.com/elasticbeanstalk/
Creative Commons Attribution Share Alike 4.0 International
283 stars 11 forks source link

Bundle commands on AL2 takes too long to complete #233

Open puneetpandey opened 2 years ago

puneetpandey commented 2 years ago

Community Note

Tell us about your request Bundler commands like bundle install and rake assets:precompile takes too long to complete

Is this request specific to an Elastic Beanstalk platform? If so, which one(s)? 64bit Amazon Linux 2 v3.4.2 running Ruby 2.7

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I am trying to migrate my Rails application from Amazon Linux AMI to Amazon Linux 2 and figured out that commands like: bundle install --local and rake assets:precompile takes too long to complete.

Are you currently working around this issue? No.

Additional context Attaching logs for more information

 % eb ssh developmentWeb
INFO: Running ssh ***FURTHER INFO TRUNCATED***
Last login: ***FURTHER INFO TRUNCATED***
  _____ _           _   _      ____                       _        _ _
 | ____| | __   ___| |_(_) ___| __ )  ___  __ _ _ __  ___| |_ __ _| | | __
 |  _| | |/ _ \/ __| __| |/ __|  _ \ / _ \/ _\ | '_ \/ __| __/ _\ | | |/ /
 | |___| | (_| \__ \ |_| | (__| |_) |  __/ (_| | | | \__ \ || (_| | |   <
 |_____|_|\__,_|___/\__|_|\___|____/ \___|\__,_|_| |_|___/\__\__,_|_|_|\_\

 Amazon Linux 2 AMI

 This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via SSH
 WILL BE LOST if the instance is replaced by auto-scaling. For more information
 on customizing your Elastic Beanstalk environment, see our documentation here:
 http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html

-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
[ec2-user@ip ~]$ tail -f /var/log/eb-engine.log 
2022/03/01 14:42:54.022791 [INFO] Executing instruction: stage ruby application
2022/03/01 14:42:54.022798 [INFO] stage ruby application ....
2022/03/01 14:42:54.022917 [INFO] Running command /bin/sh -c rbenv global
2022/03/01 14:42:54.290874 [INFO] 2.7.5

2022/03/01 14:42:54.290913 [INFO] Running command /bin/sh -c bundle --version
2022/03/01 14:42:56.243660 [INFO] Bundler version 2.3.6

2022/03/01 14:42:56.243710 [INFO] Running command /bin/sh -c bundle config set --local deployment true
2022/03/01 14:42:58.569774 [INFO] Running command /bin/sh -c bundle _2.3.6_ install --local
2022/03/01 15:05:26.285764 [INFO] Installing rake 13.0.3
Installing concurrent-ruby 1.1.8
Installing minitest 5.14.4
Installing zeitwerk 2.4.2
Installing builder 3.2.4
Installing erubi 1.10.0
Installing mini_portile2 2.5.1
Installing racc 1.5.2 with native extensions
Installing crass 1.0.6
Installing rack 2.2.3
Installing nio4r 2.5.7 with native extensions
Installing websocket-extensions 0.1.5
Installing marcel 1.0.1
Installing mini_mime 1.0.3
Installing aws-eventstream 1.1.1
Installing aws-partitions 1.453.0
Installing jmespath 1.4.0
Using bundler 2.3.6
Installing method_source 1.0.0
Installing thor 1.1.0
Installing jsonapi-renderer 0.2.2
Installing public_suffix 4.0.6
Installing amazon_pay 2.6.1
Installing ast 2.4.2
Installing execjs 2.7.0
Installing faraday-excon 1.1.0
Installing faraday-net_http 1.0.1
Installing faraday-net_http_persistent 1.1.0
Installing multipart-post 2.1.1
Installing ruby2_keywords 0.0.4
Installing multi_json 1.15.0
Installing awesome_print 1.9.2
Installing bcrypt 3.1.16 with native extensions
Installing msgpack 1.4.2 with native extensions
Installing bootstrap-filestyle-rails 1.2.1
Installing ffi 1.15.0 with native extensions
Installing libxml-ruby 3.2.1 with native extensions
Installing browser 5.3.1
Installing climate_control 0.2.0
Installing coderay 1.1.3
Installing coffee-script-source 1.12.2
Installing country-select 1.2.1
Installing dalli 2.7.11
Installing declarative 0.0.20
Installing orm_adapter 0.5.0
Installing diff-lcs 1.4.4
Installing hashie 4.1.0
Installing elasticsearch-rails 7.1.1
Installing excon 0.81.0
Installing flutie 2.2.0
Installing formatador 0.2.5
Installing mime-types-data 3.2021.0225
Installing ipaddress 0.8.3
Installing foundation_emails 2.2.1.0
Installing geocoder 1.6.7
Installing jwt 2.2.3
Installing memoist 0.16.2
Installing os 1.0.1
Installing httpclient 2.8.3
Installing trailblazer-option 0.1.1
Installing uber 0.1.0
Installing retriable 3.1.2
Installing rexml 3.2.5
Installing webrick 1.7.0
Installing sexp_processor 4.15.2
Installing temple 0.8.2
Installing tilt 2.0.10
Installing honeybadger 4.8.0
Installing htmlentities 4.3.4
Installing rainbow 3.0.0
Installing jquery-datetimepicker-rails 2.4.1.0
Installing rb-fsevent 0.10.4
Installing json 1.8.6 with native extensions
Installing kaminari-core 1.2.1
Installing memcachier 0.0.2
Installing mocha 1.12.0
Installing newrelic_rpm 7.0.0
Installing parallel 1.20.1
Installing pg 1.2.3 with native extensions
Installing rails_serve_static_assets 0.0.5
Installing rails_stdout_logging 0.0.5
Installing rdoc 4.3.0
Installing regexp_parser 2.1.1
Installing rspec-support 3.10.2
Installing ruby-progressbar 1.11.0
Installing unicode-display_width 2.0.0
Installing ruby-ole 1.2.12.2
Installing rubyzip 2.3.0
Installing slack-notifier 2.4.0
Installing turbolinks-source 5.2.0
Installing wkhtmltopdf-binary 0.12.6
Installing sitemap_generator 6.1.2
Installing i18n 1.8.10
Installing tzinfo 2.0.4
Installing rack-test 1.1.0
Installing sprockets 3.7.2
Installing warden 1.2.9
Installing font_assets 0.1.14
Installing rack-cors 1.1.1
Installing rack-maintenance 2.1.0
Installing rack-proxy 0.7.0
Installing nokogiri 1.11.3 with native extensions
Installing websocket-driver 0.7.3 with native extensions
Installing mail 2.7.1
Installing aws-sigv4 1.2.3
Installing addressable 2.7.0
Installing parser 3.0.1.1
Installing autoprefixer-rails 10.2.4.0
Installing uglifier 4.2.0
Installing faraday 1.4.1
Installing elasticsearch-api 7.12.0
Installing puma 5.6.2 with native extensions
Installing bootsnap 1.4.9 with native extensions
Installing braintree 3.4.0
Installing terrapin 0.6.0
Installing pry 0.14.1
Installing coffee-script 2.4.1
Installing mime-types 3.3.1
Installing representable 3.1.1
Installing ruby_parser 3.15.1
Installing haml 5.2.1
Installing httplog 1.4.3
Installing recaptcha 5.7.0
Installing sendgrid 1.2.4
Installing sassc 2.4.0 with native extensions
Installing get_process_mem 0.2.7
Installing rb-inotify 0.10.1
Installing rails_12factor 0.0.3
Installing sdoc 0.4.2
Installing rspec-core 3.10.1
Installing rspec-expectations 3.10.1
Installing rspec-mocks 3.10.2
Installing spreadsheet 1.2.8
Installing turbolinks 5.2.1
Installing faker 2.17.0
Installing activesupport 6.1.3.2
Installing rack-reverse-proxy 0.12.0
Installing aws-sdk-core 3.114.0
Installing css_parser 1.9.0
Installing launchy 2.5.0
Installing rubocop-ast 1.5.0
Installing faraday_middleware 1.0.0
Installing elasticsearch-transport 7.12.0
Installing signet 0.15.0
Installing pry-rails 0.3.9
Installing fog-core 2.2.4
Installing ruby2ruby 2.4.4
Installing loofah 2.9.1
Installing inky-rb 1.3.8.0
Installing mimemagic 0.3.10 with native extensions
Installing puma_worker_killer 0.3.1
Installing sass-listen 4.0.0
Installing listen 3.5.1
Installing rails-dom-testing 2.0.3
Installing globalid 0.4.2
Installing activemodel 6.1.3.2
Installing case_transform 0.2
Installing factory_bot 6.1.0
Installing jbuilder 2.11.2
Installing rack-tracker 1.12.1
Installing view_component 2.31.1
Installing wicked_pdf 2.1.0
Installing aws-sdk-sqs 1.38.0
Installing aws-sdk-kms 1.43.0
Installing premailer 1.14.3
Installing letter_opener 1.7.0
Installing rubocop 1.14.0
Installing avatax 21.3.1
Installing elasticsearch 7.12.0
Installing googleauth 0.16.2
Installing fog-json 1.2.0
Installing fog-xml 0.1.3
Installing rails-html-sanitizer 1.3.0
Installing sass 3.7.4
Installing activejob 6.1.3.2
Installing activerecord 6.1.3.2
Installing paperclip 6.0.0
Installing aws-sdk-s3 1.94.1
Installing rubocop-rails 2.9.1
Installing rubocop-rspec 2.3.0
Installing elasticsearch-model 7.1.1
Installing google-apis-core 0.3.0
Installing fog-aws 3.10.0
Installing actionview 6.1.3.2
Installing audited 4.10.0
Installing friendly_id 5.4.2
Installing hairtrigger 0.2.24
Installing kaminari-activerecord 1.2.1
Installing pg_search 2.3.5
Installing pghero 2.8.1
Installing paperclip-compression 1.1.1
Installing elasticsearch-persistence 7.1.1
Installing google-apis-analyticsreporting_v4 0.2.0
Installing actionpack 6.1.3.2
Installing kaminari-actionview 1.2.1
Installing actioncable 6.1.3.2
Installing activestorage 6.1.3.2
Installing actionmailer 6.1.3.2
Installing actionpack-action_caching 1.2.2
Installing railties 6.1.3.2
Installing sprockets-rails 3.2.2
Installing active_model_serializers 0.10.12
Installing simple_form 5.1.0
Installing kaminari 1.2.1
Installing actionmailbox 6.1.3.2
Installing actiontext 6.1.3.2
Installing premailer-rails 1.11.1
Installing coffee-rails 5.0.0
Installing responders 3.0.1
Installing factory_bot_rails 6.1.0
Installing jquery-rails 4.4.0
Installing jquery-fileupload-rails 0.4.7
Installing jquery-ui-rails 5.0.3
Installing letter_opener_web 1.4.0
Installing masonry-rails 0.2.4
Installing rails-healthcheck 1.3.0
Installing rspec-rails 4.0.2
Installing sass-rails 5.1.0
Installing rails 6.1.3.2
Installing devise 4.8.0
Installing jhtmlarea 0.1.0
Installing active_elastic_job 3.0.0
Installing apipie-rails 0.5.18
Installing chartjs-ror 3.6.4
Installing dropzonejs-rails 0.8.5

2022/03/01 15:05:26.285839 [INFO] Running command /bin/sh -c rbenv rehash
2022/03/01 15:05:26.607177 [INFO] encountered a Gemfile.lock, setting proper permissions to user webapp
2022/03/01 15:05:26.607721 [INFO] Executing instruction: check default web server
2022/03/01 15:05:26.611915 [INFO] Executing instruction: check procfile
2022/03/01 15:05:26.611949 [INFO] creating default Procfile...
2022/03/01 15:05:26.612013 [INFO] Executing instruction: configure X-Ray
2022/03/01 15:05:26.612023 [INFO] X-Ray is not enabled.
2022/03/01 15:05:26.612028 [INFO] Executing instruction: Configure proxy Nginx
2022/03/01 15:05:26.612032 [INFO] configuring proxy...
2022/03/01 15:05:26.612355 [INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/webapp.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/webapp.conf
2022/03/01 15:05:26.613890 [INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/elasticbeanstalk-nginx-ruby-upstream.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk-nginx-ruby-upstream.conf
2022/03/01 15:05:26.621599 [INFO] Executing instruction: configure healthd specific proxy conf
2022/03/01 15:05:26.622983 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/01 15:05:26.630524 [WARN] Warning: process healthd is already registered...
Deregistering the process ...
2022/03/01 15:05:26.630566 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/01 15:05:26.642163 [INFO] Running command /bin/sh -c systemctl is-active healthd.service
2022/03/01 15:05:26.656763 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/01 15:05:26.670675 [INFO] Running command /bin/sh -c systemctl stop healthd.service
2022/03/01 15:05:26.716688 [INFO] Running command /bin/sh -c systemctl disable healthd.service
2022/03/01 15:05:26.880532 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/01 15:05:27.061902 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/01 15:05:27.078915 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/01 15:05:27.269075 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/01 15:05:27.277403 [INFO] Running command /bin/sh -c systemctl is-enabled aws-eb.target
2022/03/01 15:05:27.289611 [INFO] Running command /bin/sh -c systemctl enable aws-eb.target
2022/03/01 15:05:27.467472 [INFO] Running command /bin/sh -c systemctl start aws-eb.target
2022/03/01 15:05:27.485135 [INFO] Running command /bin/sh -c systemctl enable healthd.service
2022/03/01 15:05:27.663103 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/01 15:05:27.678017 [INFO] Running command /bin/sh -c systemctl is-active healthd.service
2022/03/01 15:05:27.698044 [INFO] Running command /bin/sh -c systemctl start healthd.service
2022/03/01 15:05:27.740271 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_logformat.conf to /var/proxy/staging/nginx/conf.d/healthd_logformat.conf
2022/03/01 15:05:27.741490 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_nginx.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/healthd.conf
2022/03/01 15:05:27.742771 [INFO] Executing instruction: rake tasks
2022/03/01 15:05:27.742807 [INFO] Running command /bin/sh -c npm bin --global
2022/03/01 15:05:28.253180 [INFO] /opt/elasticbeanstalk/support/node-install/node-v14.18.3-linux-x64/bin
2022/03/01 15:05:28.253226 [INFO] Running command /bin/sh -c rbenv global
2022/03/01 15:05:28.370704 [INFO] 2.7.5
2022/03/01 15:05:28.370749 [INFO] Running command /bin/su webapp -c bundle exec /opt/elasticbeanstalk/config/private/checkforraketask.rb assets:precompile
2022/03/01 15:05:50.883461 [INFO] Found assets:precompile task in Rakefile
2022/03/01 15:05:50.883526 [INFO] Running command /bin/su webapp -c bundle exec rake assets:precompile
2022/03/01 15:15:56.386967 [INFO] yarn install v1.22.17
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 19.40s.

Couple of things here:

  1. bundle install --local took 23+ mins to execute and rake assets:precompile took 10+ minutes to execute.
  2. I am using CodePipeline to trigger the deployment. So every time the process gets stuck on bundle install for more than x minutes, I see build failed (see attached)
  3. I am seeing this message at the end
2022/03/01 15:17:34.111920 [INFO] Executing instruction: support rails logging
2022/03/01 15:17:34.111933 [INFO] adding builtin Rails logging support
2022/03/01 15:17:34.123753 [INFO] create soft link from /var/app/current/log/development.log to /var/app/containerfiles/logs/development.log
2022/03/01 15:17:34.123811 [INFO] Executing instruction: RunAppDeployPostDeployHooks
2022/03/01 15:17:34.123827 [INFO] Executing platform hooks in .platform/hooks/postdeploy/
2022/03/01 15:17:34.123896 [INFO] Following scripts will be executed in order: [01_set_file_permissions.sh]
2022/03/01 15:17:34.123904 [INFO] Running script: .platform/hooks/postdeploy/01_set_file_permissions.sh
2022/03/01 15:17:34.534842 [INFO] Finished running scripts in /var/app/current/.platform/hooks/postdeploy
2022/03/01 15:17:34.534870 [INFO] Executing cleanup logic
2022/03/01 15:17:34.534981 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment completed successfully.","timestamp":1646147854,"severity":"INFO"}]}]}

2022/03/01 15:17:34.535176 [INFO] Platform Engine finished execution on command: app-deploy

But when I am doing bundle exec rails console, I am getting this error:

/var/app/current/config/environments/development.rb:79:inblock in

': undefined method []' for nil:NilClass (NoMethodError)

and I have this on line#79: Rails.application.credentials.aws[:s3_access_key]

So, it looks to me that Environment variables are not loaded properly.

Screenshot 2022-03-01 at 8 53 42 PM

Please suggest!

puneetpandey commented 2 years ago

I followed this link: https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/ and I am able to see env variables - when not logged-in as a root user. When I shift to root user, I don't see a value of any of the env variable and rails console problem stays as-is.

puneetpandey commented 2 years ago

More updates:

  1. I observed that when I launch a new AL2 EC2 instance the initial time taken by bundle install during launch is 2-4 minutes and then further deployments start taking more and more time (like 6 minutes.. 14 minutes and so on..).
  2. I kept same bundler version 2.3.6 but no changes in deployment time and rails console
nickpearson commented 2 years ago

Are you seeing any warnings about CPU and/or memory? I've observed something similar when I tried using an instance size that was too small to compile native gems and run assets:precompile while also running my app. The fact that the app isn't running yet during the initial deployment but is running thereafter may explain why bundler and precompilation take so much longer on subsequent deploys.

You might also try setting the BUNDLER_DEPLOYMENT_MODE environment property to false to allow bundler to cache your gems, meaning on subsequent deploys it will only have to install new/updated gems. (Be sure to understand the implications of this change first, and note that it won't help with the asset precompilation step.)

Regarding your question about rails console, I wrote a short script that makes the Elastic Beanstalk environment properties available as shell environment variables on a per-command basis, which you can find here. I have a hook script that copies this into /usr/local/bin so that it can be called from anywhere, like this:

cd /var/app/current
ebenv bin/rails c
puneetpandey commented 2 years ago

Hi @nickpearson Thank you for your reply

I am using t2.medium EC2 instance - this worked for my application running on AMI 64bit Amazon Linux 2018.03 v2.12.10 running Ruby 2.6 (Puma) But I have upgraded this instance to t2.large now and I do see that the time taken by bundle install is now reduced but it gets increased with subsequent deploys but this has not reached 20+ mins as of yet - so I think I can still improve this. Somehow I feel that some gem / some process is causing bundle install to delay.

Do you know if I can add --verbose to bundle install when deployment starts? Or any log file where I can see what bundle install is trying to do OR the in-depth logs for eb-engine.log

I also ran your shell script and I can access ENV variables as a non-root user but as a root-user I still can't access.

nickpearson commented 2 years ago

I would expect t2.medium to have enough memory, but it depends on how much your app is consuming at the time that bundle install runs. I don't know if it's possible to use --verbose during a deployment, but you could always run bundler manually in verbose mode to see if you can tell where the slowdown is. BUNDLER_DEPLOYMENT_MODE may be your friend here if you don't need a fresh install of every gem on every deploy.

I just tried my script (which gets auto-installed to /usr/local/bin/ebenv using a prebuild hook) and it works when run as root. For example, ebenv echo '$RACK_ENV' outputs production for me when run as root. (If you inspect variables like this, you need to single-quote them or escape the $ as \$ to prevent shell expansion before the arguments get passed to ebenv.)

The script doesn't handle root vs non-root differently. It just looks up the app user (/opt/elasticbeanstalk/bin/get-config platformconfig -k AppUser which should return "webapp") and reads the /opt/elasticbeanstalk/deployment/env file, then runs the specified command as the webapp user. I don't know why it wouldn't work for you when run as root, unless you're trying to run a command that the webapp user isn't allowed to run.

puneetpandey commented 2 years ago

Hi @nickpearson thank you again for helping me on this. Appreciate your help I am having the same expectation that t2.medium to have enough memory but I am going to troubleshoot it with your suggestion to append --verbose manually.

I placed your script under .platform/hooks/prebuild/03_set_env_vars.sh and then deployed the application. I am attaching the logs for your reference and what I tried after the deployment finished.

2022/03/08 12:34:41.559348 [INFO] Starting...
2022/03/08 12:34:41.559412 [INFO] Starting EBPlatform-PlatformEngine
2022/03/08 12:34:41.559428 [INFO] reading event message file
2022/03/08 12:34:41.559532 [INFO] no eb envtier info file found, skip loading env tier info.
2022/03/08 12:34:41.559605 [INFO] Engine received EB command cfn-hup-exec
2022/03/08 12:34:41.683222 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBAutoScalingGroup --region us-east-1
2022/03/08 12:34:42.224940 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBBeanstalkMetadata --region us-east-1
2022/03/08 12:34:42.827205 [INFO] checking whether command app-deploy is applicable to this instance...
2022/03/08 12:34:42.827219 [INFO] this command is applicable to the instance, thus instance should execute command
2022/03/08 12:34:42.827223 [INFO] Engine command: (app-deploy)

2022/03/08 12:34:42.827230 [INFO] Downloading EB Application...
2022/03/08 12:34:42.827234 [INFO] Region: us-east-1
2022/03/08 12:34:42.827238 [INFO] envID: XXXXXXXXXXXXX
2022/03/08 12:34:42.827242 [INFO] envBucket: elasticbeanstalk-us-east-1
2022/03/08 12:34:42.827246 [INFO] Using manifest file name from command request
2022/03/08 12:34:42.827253 [INFO] Manifest name is : manifest
2022/03/08 12:34:42.827257 [INFO] Download app version manifest
2022/03/08 12:34:42.827369 [INFO] Downloading: bucket: elasticbeanstalk-us-east-1, object: /resources/environments/XXXXXXXXXXXXX/_runtime/versions/manifest
2022/03/08 12:34:42.868359 [INFO] Download successful170bytes downloaded
2022/03/08 12:34:42.868484 [INFO] Trying to read and parse version manifest...
2022/03/08 12:34:42.870338 [INFO] Downloading: bucket: elasticbeanstalk-us-east-1, object: /resources/environments/XXXXXXXXXXXX/_runtime/_versions/application/code-pipeline
2022/03/08 12:34:43.192622 [INFO] Download successful14807845bytes downloaded
2022/03/08 12:34:43.192835 [INFO] Executing instruction: ElectLeader
2022/03/08 12:34:43.192842 [INFO] Running leader election for instance i-04ce15ff4e1b3e99f...
2022/03/08 12:34:43.192846 [INFO] Calling the cfn-elect-cmd-leader to elect the command leader.
2022/03/08 12:34:43.192859 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-elect-cmd-leader --stack arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd --command-name ElasticBeanstalkCommand-AWSEBAutoScalingGroup --invocation-id 0d064685-2768-4e22-98de-3b2caeff75fe --listener-id i-04ce15ff4e1b3e99f --region us-east-1
2022/03/08 12:34:43.768376 [INFO] Instance is Leader.
2022/03/08 12:34:43.768440 [INFO] Executing instruction: stopSqsd
2022/03/08 12:34:43.768448 [INFO] This is a web server environment instance, skip stop sqsd daemon ...
2022/03/08 12:34:43.768453 [INFO] Executing instruction: PreBuildEbExtension
2022/03/08 12:34:43.768458 [INFO] Starting executing the config set Infra-EmbeddedPreBuild.
2022/03/08 12:34:43.768491 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBAutoScalingGroup --region us-east-1 --configsets Infra-EmbeddedPreBuild
2022/03/08 12:34:44.212332 [INFO] Finished executing the config set Infra-EmbeddedPreBuild.
2022/03/08 12:34:44.212358 [INFO] Executing instruction: StageApplication
2022/03/08 12:34:44.214467 [INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/
2022/03/08 12:34:44.214486 [INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/
2022/03/08 12:34:44.736312 [INFO] finished extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/ successfully
2022/03/08 12:34:44.771641 [INFO] Executing instruction: RunAppDeployPreBuildHooks
2022/03/08 12:34:44.771668 [INFO] Executing platform hooks in .platform/hooks/prebuild/
2022/03/08 12:34:44.771728 [INFO] Following scripts will be executed in order: [01_install_git.sh 02_install_postgres.sh 03_set_env_vars.sh]
2022/03/08 12:34:44.771735 [INFO] Running script: .platform/hooks/prebuild/01_install_git.sh
2022/03/08 12:34:47.957724 [INFO] Running script: .platform/hooks/prebuild/02_install_postgres.sh
2022/03/08 12:34:47.985415 [INFO] Running script: .platform/hooks/prebuild/03_set_env_vars.sh
2022/03/08 12:34:49.347155 [INFO] Finished running scripts in /var/app/staging/.platform/hooks/prebuild
2022/03/08 12:34:49.347177 [INFO] Executing instruction: stage ruby application
2022/03/08 12:34:49.347183 [INFO] stage ruby application ....
2022/03/08 12:34:49.347250 [INFO] Running command /bin/sh -c rbenv global
2022/03/08 12:34:49.461085 [INFO] 2.7.5
2022/03/08 12:34:49.461120 [INFO] Running command /bin/sh -c bundle --version
2022/03/08 12:34:49.963514 [INFO] Bundler version 2.3.7
2022/03/08 12:34:49.963547 [INFO] Running command /bin/sh -c bundle config set --local deployment true
2022/03/08 12:34:50.210999 [INFO] Running command /bin/sh -c bundle _2.3.7_ install
2022/03/08 12:40:07.539687 [INFO] Fetching gem metadata from https://rubygems.org/......
Fetching https://github.com/GuillaumeWrobel/gridhook.git
Fetching https://github.com/waynehoover/s3_direct_upload.git
Fetching rake 13.0.3
Installing rake 13.0.3
Fetching concurrent-ruby 1.1.8
Fetching minitest 5.14.4
Installing minitest 5.14.4
Installing concurrent-ruby 1.1.8
Fetching zeitwerk 2.4.2
Installing zeitwerk 2.4.2
Fetching builder 3.2.4
Installing builder 3.2.4
Fetching erubi 1.10.0
Fetching mini_portile2 2.5.1
Installing erubi 1.10.0
Installing mini_portile2 2.5.1
Fetching racc 1.5.2
Fetching crass 1.0.6
Installing crass 1.0.6
Installing racc 1.5.2 with native extensions
Fetching rack 2.2.3
Installing rack 2.2.3
Fetching nio4r 2.5.7
Installing nio4r 2.5.7 with native extensions
Fetching websocket-extensions 0.1.5
Installing websocket-extensions 0.1.5
Fetching marcel 1.0.1
Installing marcel 1.0.1
Fetching mini_mime 1.0.3
Installing mini_mime 1.0.3
Fetching aws-eventstream 1.1.1
Installing aws-eventstream 1.1.1
Fetching aws-partitions 1.453.0
Installing aws-partitions 1.453.0
Fetching jmespath 1.4.0
Installing jmespath 1.4.0
Using bundler 2.3.7
Fetching method_source 1.0.0
Installing method_source 1.0.0
Fetching thor 1.1.0
Installing thor 1.1.0
Fetching jsonapi-renderer 0.2.2
Installing jsonapi-renderer 0.2.2
Fetching public_suffix 4.0.6
Installing public_suffix 4.0.6
Fetching amazon_pay 2.6.1
Installing amazon_pay 2.6.1
Fetching execjs 2.7.0
Installing execjs 2.7.0
Fetching faraday-excon 1.1.0
Installing faraday-excon 1.1.0
Fetching faraday-net_http 1.0.1
Installing faraday-net_http 1.0.1
Fetching faraday-net_http_persistent 1.1.0
Installing faraday-net_http_persistent 1.1.0
Fetching multipart-post 2.1.1
Installing multipart-post 2.1.1
Fetching ruby2_keywords 0.0.4
Installing ruby2_keywords 0.0.4
Fetching multi_json 1.15.0
Installing multi_json 1.15.0
Fetching bcrypt 3.1.16
Installing bcrypt 3.1.16 with native extensions
Fetching msgpack 1.4.2
Installing msgpack 1.4.2 with native extensions
Fetching bootstrap-filestyle-rails 1.2.1
Installing bootstrap-filestyle-rails 1.2.1
Fetching ffi 1.15.0
Installing ffi 1.15.0 with native extensions
Fetching libxml-ruby 3.2.1
Installing libxml-ruby 3.2.1 with native extensions
Fetching browser 5.3.1
Installing browser 5.3.1
Fetching climate_control 0.2.0
Installing climate_control 0.2.0
Fetching coderay 1.1.3
Installing coderay 1.1.3
Fetching coffee-script-source 1.12.2
Installing coffee-script-source 1.12.2
Fetching country-select 1.2.1
Installing country-select 1.2.1
Fetching dalli 2.7.11
Installing dalli 2.7.11
Fetching declarative 0.0.20
Installing declarative 0.0.20
Fetching orm_adapter 0.5.0
Installing orm_adapter 0.5.0
Fetching hashie 4.1.0
Installing hashie 4.1.0
Fetching elasticsearch-rails 7.1.1
Installing elasticsearch-rails 7.1.1
Fetching excon 0.81.0
Installing excon 0.81.0
Fetching flutie 2.2.0
Installing flutie 2.2.0
Fetching formatador 0.2.5
Installing formatador 0.2.5
Fetching mime-types-data 3.2021.0225
Installing mime-types-data 3.2021.0225
Fetching ipaddress 0.8.3
Installing ipaddress 0.8.3
Fetching foundation_emails 2.2.1.0
Installing foundation_emails 2.2.1.0
Fetching geocoder 1.6.7
Installing geocoder 1.6.7
Fetching jwt 2.2.3
Installing jwt 2.2.3
Fetching memoist 0.16.2
Installing memoist 0.16.2
Fetching os 1.0.1
Installing os 1.0.1
Fetching httpclient 2.8.3
Installing httpclient 2.8.3
Fetching trailblazer-option 0.1.1
Installing trailblazer-option 0.1.1
Fetching uber 0.1.0
Installing uber 0.1.0
Fetching retriable 3.1.2
Installing retriable 3.1.2
Fetching rexml 3.2.5
Installing rexml 3.2.5
Fetching webrick 1.7.0
Installing webrick 1.7.0
Fetching sexp_processor 4.15.2
Installing sexp_processor 4.15.2
Fetching temple 0.8.2
Installing temple 0.8.2
Fetching tilt 2.0.10
Installing tilt 2.0.10
Fetching honeybadger 4.8.0
Installing honeybadger 4.8.0
Fetching htmlentities 4.3.4
Installing htmlentities 4.3.4
Fetching jquery-datetimepicker-rails 2.4.1.0
Installing jquery-datetimepicker-rails 2.4.1.0
Fetching rb-fsevent 0.10.4
Installing rb-fsevent 0.10.4
Fetching json 1.8.6
Installing json 1.8.6 with native extensions
Fetching kaminari-core 1.2.1
Installing kaminari-core 1.2.1
Fetching memcachier 0.0.2
Installing memcachier 0.0.2
Fetching newrelic_rpm 7.0.0
Installing newrelic_rpm 7.0.0
Fetching pg 1.2.3
Installing pg 1.2.3 with native extensions
Fetching rails_serve_static_assets 0.0.5
Installing rails_serve_static_assets 0.0.5
Fetching rails_stdout_logging 0.0.5
Installing rails_stdout_logging 0.0.5
Fetching rdoc 4.3.0
Installing rdoc 4.3.0
Fetching ruby-ole 1.2.12.2
Installing ruby-ole 1.2.12.2
Fetching rubyzip 2.3.0
Installing rubyzip 2.3.0
Fetching slack-notifier 2.4.0
Installing slack-notifier 2.4.0
Fetching turbolinks-source 5.2.0
Installing turbolinks-source 5.2.0
Fetching wkhtmltopdf-binary 0.12.6
Fetching i18n 1.8.10
Installing i18n 1.8.10
Fetching tzinfo 2.0.4
Installing tzinfo 2.0.4
Fetching sitemap_generator 6.1.2
Installing sitemap_generator 6.1.2
Fetching rack-test 1.1.0
Installing rack-test 1.1.0
Fetching sprockets 3.7.2
Installing sprockets 3.7.2
Fetching warden 1.2.9
Installing warden 1.2.9
Fetching font_assets 0.1.14
Installing font_assets 0.1.14
Fetching rack-cors 1.1.1
Installing rack-cors 1.1.1
Fetching rack-maintenance 2.1.0
Installing rack-maintenance 2.1.0
Fetching rack-proxy 0.7.0
Installing rack-proxy 0.7.0
Fetching nokogiri 1.11.3 (x86_64-linux)
Installing nokogiri 1.11.3 (x86_64-linux)
Fetching websocket-driver 0.7.3
Installing websocket-driver 0.7.3 with native extensions
Fetching mail 2.7.1
Installing mail 2.7.1
Fetching aws-sigv4 1.2.3
Installing aws-sigv4 1.2.3
Fetching addressable 2.7.0
Installing addressable 2.7.0
Fetching autoprefixer-rails 10.2.4.0
Installing autoprefixer-rails 10.2.4.0
Fetching uglifier 4.2.0
Installing uglifier 4.2.0
Fetching faraday 1.4.1
Installing faraday 1.4.1
Fetching elasticsearch-api 7.12.0
Installing elasticsearch-api 7.12.0
Fetching puma 5.6.2
Installing puma 5.6.2 with native extensions
Installing wkhtmltopdf-binary 0.12.6
Fetching bootsnap 1.4.9
Installing bootsnap 1.4.9 with native extensions
Fetching braintree 3.4.0
Installing braintree 3.4.0
Fetching terrapin 0.6.0
Installing terrapin 0.6.0
Fetching pry 0.14.1
Installing pry 0.14.1
Fetching coffee-script 2.4.1
Installing coffee-script 2.4.1
Fetching mime-types 3.3.1
Installing mime-types 3.3.1
Fetching representable 3.1.1
Installing representable 3.1.1
Fetching ruby_parser 3.15.1
Installing ruby_parser 3.15.1
Fetching haml 5.2.1
Installing haml 5.2.1
Fetching recaptcha 5.7.0
Installing recaptcha 5.7.0
Fetching sendgrid 1.2.4
Installing sendgrid 1.2.4
Fetching rails_12factor 0.0.3
Installing rails_12factor 0.0.3
Fetching sdoc 0.4.2
Installing sdoc 0.4.2
Fetching spreadsheet 1.2.8
Installing spreadsheet 1.2.8
Fetching turbolinks 5.2.1
Installing turbolinks 5.2.1
Fetching sassc 2.4.0
Installing sassc 2.4.0 with native extensions
Fetching get_process_mem 0.2.7
Installing get_process_mem 0.2.7
Fetching rb-inotify 0.10.1
Installing rb-inotify 0.10.1
Fetching activesupport 6.1.3.2
Installing activesupport 6.1.3.2
Fetching rack-reverse-proxy 0.12.0
Installing rack-reverse-proxy 0.12.0
Fetching loofah 2.9.1
Installing loofah 2.9.1
Fetching inky-rb 1.3.8.0
Installing inky-rb 1.3.8.0
Fetching mimemagic 0.3.10
Installing mimemagic 0.3.10 with native extensions
Fetching aws-sdk-core 3.114.0
Installing aws-sdk-core 3.114.0
Fetching css_parser 1.9.0
Installing css_parser 1.9.0
Fetching faraday_middleware 1.0.0
Installing faraday_middleware 1.0.0
Fetching elasticsearch-transport 7.12.0
Installing elasticsearch-transport 7.12.0
Fetching signet 0.15.0
Installing signet 0.15.0
Fetching pry-rails 0.3.9
Installing pry-rails 0.3.9
Fetching fog-core 2.2.4
Installing fog-core 2.2.4
Fetching ruby2ruby 2.4.4
Installing ruby2ruby 2.4.4
Fetching puma_worker_killer 0.3.1
Installing puma_worker_killer 0.3.1
Fetching sass-listen 4.0.0
Installing sass-listen 4.0.0
Fetching rails-dom-testing 2.0.3
Installing rails-dom-testing 2.0.3
Fetching globalid 0.4.2
Installing globalid 0.4.2
Fetching activemodel 6.1.3.2
Installing activemodel 6.1.3.2
Fetching case_transform 0.2
Installing case_transform 0.2
Fetching jbuilder 2.11.2
Installing jbuilder 2.11.2
Fetching rack-tracker 1.12.1
Installing rack-tracker 1.12.1
Fetching view_component 2.31.1
Installing view_component 2.31.1
Fetching wicked_pdf 2.1.0
Installing wicked_pdf 2.1.0
Fetching rails-html-sanitizer 1.3.0
Installing rails-html-sanitizer 1.3.0
Fetching aws-sdk-sqs 1.38.0
Installing aws-sdk-sqs 1.38.0
Fetching aws-sdk-kms 1.43.0
Installing aws-sdk-kms 1.43.0
Fetching premailer 1.14.3
Installing premailer 1.14.3
Fetching avatax 21.3.1
Installing avatax 21.3.1
Fetching elasticsearch 7.12.0
Installing elasticsearch 7.12.0
Fetching googleauth 0.16.2
Installing googleauth 0.16.2
Fetching fog-json 1.2.0
Installing fog-json 1.2.0
Fetching fog-xml 0.1.3
Installing fog-xml 0.1.3
Fetching sass 3.7.4
Installing sass 3.7.4
Fetching activejob 6.1.3.2
Installing activejob 6.1.3.2
Fetching activerecord 6.1.3.2
Installing activerecord 6.1.3.2
Fetching paperclip 6.0.0
Installing paperclip 6.0.0
Fetching actionview 6.1.3.2
Installing actionview 6.1.3.2
Fetching aws-sdk-s3 1.94.1
Installing aws-sdk-s3 1.94.1
Fetching elasticsearch-model 7.1.1
Installing elasticsearch-model 7.1.1
Fetching google-apis-core 0.3.0
Installing google-apis-core 0.3.0
Fetching fog-aws 3.10.0
Installing fog-aws 3.10.0
Fetching audited 4.10.0
Installing audited 4.10.0
Fetching friendly_id 5.4.2
Installing friendly_id 5.4.2
Fetching hairtrigger 0.2.24
Installing hairtrigger 0.2.24
Fetching kaminari-activerecord 1.2.1
Installing kaminari-activerecord 1.2.1
Fetching pg_search 2.3.5
Installing pg_search 2.3.5
Fetching pghero 2.8.1
Installing pghero 2.8.1
Fetching paperclip-compression 1.1.1
Installing paperclip-compression 1.1.1
Fetching actionpack 6.1.3.2
Installing actionpack 6.1.3.2
Fetching kaminari-actionview 1.2.1
Installing kaminari-actionview 1.2.1
Fetching elasticsearch-persistence 7.1.1
Installing elasticsearch-persistence 7.1.1
Fetching google-apis-analyticsreporting_v4 0.2.0
Installing google-apis-analyticsreporting_v4 0.2.0
Fetching actioncable 6.1.3.2
Installing actioncable 6.1.3.2
Fetching activestorage 6.1.3.2
Installing activestorage 6.1.3.2
Fetching actionmailer 6.1.3.2
Installing actionmailer 6.1.3.2
Fetching actionpack-action_caching 1.2.2
Installing actionpack-action_caching 1.2.2
Fetching railties 6.1.3.2
Installing railties 6.1.3.2
Fetching sprockets-rails 3.2.2
Installing sprockets-rails 3.2.2
Fetching active_model_serializers 0.10.12
Installing active_model_serializers 0.10.12
Fetching simple_form 5.1.0
Installing simple_form 5.1.0
Fetching kaminari 1.2.1
Installing kaminari 1.2.1
Fetching actionmailbox 6.1.3.2
Installing actionmailbox 6.1.3.2
Fetching actiontext 6.1.3.2
Installing actiontext 6.1.3.2
Fetching premailer-rails 1.11.1
Installing premailer-rails 1.11.1
Fetching coffee-rails 5.0.0
Installing coffee-rails 5.0.0
Fetching responders 3.0.1
Installing responders 3.0.1
Fetching jquery-rails 4.4.0
Installing jquery-rails 4.4.0
Fetching jquery-fileupload-rails 0.4.7
Installing jquery-fileupload-rails 0.4.7
Fetching jquery-ui-rails 5.0.3
Installing jquery-ui-rails 5.0.3
Fetching masonry-rails 0.2.4
Installing masonry-rails 0.2.4
Fetching rails-healthcheck 1.3.0
Installing rails-healthcheck 1.3.0
Fetching sass-rails 5.1.0
Installing sass-rails 5.1.0
Fetching rails 6.1.3.2
Installing rails 6.1.3.2
Fetching devise 4.8.0
Installing devise 4.8.0
Fetching jhtmlarea 0.1.0
Installing jhtmlarea 0.1.0
Fetching active_elastic_job 3.0.0
Installing active_elastic_job 3.0.0
Fetching apipie-rails 0.5.18
Installing apipie-rails 0.5.18
Fetching chartjs-ror 3.6.4
Installing chartjs-ror 3.6.4
Fetching dropzonejs-rails 0.8.5
Installing dropzonejs-rails 0.8.5
Using gridhook 0.2.1 from https://github.com/GuillaumeWrobel/gridhook.git (at master@3adac66)
Using s3_direct_upload 0.1.7 from https://github.com/waynehoover/s3_direct_upload.git (at master@6f6decc)
Fetching bootstrap-sass 3.4.1
Installing bootstrap-sass 3.4.1
Bundle complete! 90 Gemfile dependencies, 201 gems now installed.
Gems in the groups 'test' and 'development' were not installed.
Bundled gems are installed into `./vendor/bundle`
Post-install message from avatax:
********************************************************************************

AvaTax REST API
------------------------------
Our developer site documents the AvaTax REST API.
(http://developer.avatax.com).
Blog
----------------------------
The Developer Blog is a great place to learn more about the API and AvaTax integrations
Subscribe to the RSS feed be notified of new posts:
(http://developer.avatax.com/blog).

********************************************************************************
Post-install message from sass:

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

Post-install message from paperclip:
##################################################
#  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER      #
##################################################

Paperclip is now compatible with aws-sdk >= 2.0.0.

If you are using S3 storage, aws-sdk >= 2.0.0 requires you to make a few small
changes:

* You must set the `s3_region`
* If you are explicitly setting permissions anywhere, such as in an initializer,
  note that the format of the permissions changed from using an underscore to
  using a hyphen. For example, `:public_read` needs to be changed to
  `public-read`.

For a walkthrough of upgrading from 4 to 5 and aws-sdk >= 2.0 you can watch
http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5
Post-install message from rack-maintenance:

******** Rack::Maintenance 2.0.0 *********

Rack::Maintenance 2.0.0 has dropped the built-in filter for /assets.
This means that all your paths will now be blocked in maintenance
mode, unless you exclude them in the configuration.

To bring back pre-2.0.0 behaviour, configure the gem like so:

config.middleware.use 'Rack::Maintenance',
  file: Rails.root.join("public", "maintenance.html"),
  without: /\A\/assets/

2022/03/08 12:40:07.539749 [INFO] Running command /bin/sh -c rbenv rehash
2022/03/08 12:40:07.797058 [INFO] encountered a Gemfile.lock, setting proper permissions to user webapp
2022/03/08 12:40:07.797193 [INFO] Executing instruction: check default web server
2022/03/08 12:40:07.801105 [INFO] Executing instruction: check procfile
2022/03/08 12:40:07.801127 [INFO] creating default Procfile...
2022/03/08 12:40:07.801177 [INFO] Executing instruction: configure X-Ray
2022/03/08 12:40:07.801183 [INFO] X-Ray is not enabled.
2022/03/08 12:40:07.801187 [INFO] Executing instruction: Configure proxy Nginx
2022/03/08 12:40:07.801191 [INFO] configuring proxy...
2022/03/08 12:40:07.801507 [INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/webapp.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/webapp.conf
2022/03/08 12:40:07.802775 [INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/elasticbeanstalk-nginx-ruby-upstream.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk-nginx-ruby-upstream.conf
2022/03/08 12:40:07.807589 [INFO] Executing instruction: configure healthd specific proxy conf
2022/03/08 12:40:07.808826 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/08 12:40:07.826276 [WARN] Warning: process healthd is already registered...
Deregistering the process ...
2022/03/08 12:40:07.826301 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/08 12:40:07.839541 [INFO] Running command /bin/sh -c systemctl is-active healthd.service
2022/03/08 12:40:07.850828 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/08 12:40:07.866923 [INFO] Running command /bin/sh -c systemctl stop healthd.service
2022/03/08 12:40:07.918938 [INFO] Running command /bin/sh -c systemctl disable healthd.service
2022/03/08 12:40:08.074472 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:40:08.211703 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:40:08.224685 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:40:08.363449 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:40:08.376753 [INFO] Running command /bin/sh -c systemctl is-enabled aws-eb.target
2022/03/08 12:40:08.391777 [INFO] Running command /bin/sh -c systemctl enable aws-eb.target
2022/03/08 12:40:08.539821 [INFO] Running command /bin/sh -c systemctl start aws-eb.target
2022/03/08 12:40:08.548078 [INFO] Running command /bin/sh -c systemctl enable healthd.service
2022/03/08 12:40:08.690386 [INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
2022/03/08 12:40:08.700606 [INFO] Running command /bin/sh -c systemctl is-active healthd.service
2022/03/08 12:40:08.715567 [INFO] Running command /bin/sh -c systemctl start healthd.service
2022/03/08 12:40:08.784834 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_logformat.conf to /var/proxy/staging/nginx/conf.d/healthd_logformat.conf
2022/03/08 12:40:08.785924 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_nginx.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/healthd.conf
2022/03/08 12:40:08.787142 [INFO] Executing instruction: rake tasks
2022/03/08 12:40:08.787171 [INFO] Running command /bin/sh -c npm bin --global
2022/03/08 12:40:09.550029 [INFO] /opt/elasticbeanstalk/support/node-install/node-v16.14.0-linux-x64/bin
2022/03/08 12:40:09.550062 [INFO] Running command /bin/sh -c rbenv global
2022/03/08 12:40:09.599564 [INFO] 2.7.5
2022/03/08 12:40:09.599600 [INFO] Running command /bin/su webapp -c bundle exec /opt/elasticbeanstalk/config/private/checkforraketask.rb assets:precompile
2022/03/08 12:40:22.597862 [INFO] Found assets:precompile task in Rakefile
2022/03/08 12:40:22.597916 [INFO] Running command /bin/su webapp -c bundle exec rake assets:precompile
2022/03/08 12:45:14.056672 [INFO] Dalli::Server#connect mem.cache.amazonaws.com:11211
** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.8.0 framework=rails level=1 pid=15514
yarn install v1.22.17
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 1.96s.

2022/03/08 12:45:14.056715 [INFO] Running command /bin/sh -c npm bin --global
2022/03/08 12:45:14.685650 [INFO] /opt/elasticbeanstalk/support/node-install/node-v16.14.0-linux-x64/bin
2022/03/08 12:45:14.685682 [INFO] Running command /bin/sh -c rbenv global
2022/03/08 12:45:14.771744 [INFO] 2.7.5
2022/03/08 12:45:14.771779 [INFO] Running command /bin/su webapp -c bundle exec /opt/elasticbeanstalk/config/private/checkforraketask.rb db:migrate
2022/03/08 12:45:19.432816 [INFO] Found db:migrate task in Rakefile
2022/03/08 12:45:19.432877 [INFO] Running command /bin/su webapp -c bundle exec rake db:migrate
2022/03/08 12:45:25.220113 [INFO] Dalli::Server#connect mem.cache.amazonaws.com:11211
** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.8.0 framework=rails level=1 pid=16937
   (0.5ms)  SELECT pg_try_advisory_lock(6211929828156795120)
   (1.0ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC
  ActiveRecord::InternalMetadata Load (0.5ms)  SELECT "ar_internal_metadata".* FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 LIMIT $2  [["key", "environment"], ["LIMIT", 1]]
   (0.5ms)  SELECT pg_advisory_unlock(6211929828156795120)

2022/03/08 12:45:25.220141 [INFO] Executing instruction: configure log streaming
2022/03/08 12:45:25.220149 [INFO] start to get cloudwatch log client
2022/03/08 12:45:25.220299 [INFO] start to config log streaming
2022/03/08 12:45:25.220311 [INFO] start to configure log streaming config file
2022/03/08 12:45:25.220546 [INFO] start to create cloudwatch log stream
2022/03/08 12:45:25.220552 [INFO] start to create log streaming
2022/03/08 12:45:25.242698 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/eb-engine.log already exists. No need to create it.
2022/03/08 12:45:25.264922 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/eb-hooks.log already exists. No need to create it.
2022/03/08 12:45:25.285812 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/nginx/access.log already exists. No need to create it.
2022/03/08 12:45:25.305959 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/nginx/error.log already exists. No need to create it.
2022/03/08 12:45:25.327214 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/web.stdout.log already exists. No need to create it.
2022/03/08 12:45:25.363541 [WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/puma/puma.log already exists. No need to create it.
2022/03/08 12:45:25.363563 [INFO] start to fetch config
2022/03/08 12:45:25.363580 [INFO] Running command /bin/sh -c /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -c file:/opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json -s
2022/03/08 12:45:26.223843 [INFO] ****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_beanstalk.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Valid Json input schema.
I! Detecting run_as_user...
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded

2022/03/08 12:45:26.223870 [INFO] Executing instruction: GetToggleForceRotate
2022/03/08 12:45:26.223874 [INFO] Checking if logs need forced rotation
2022/03/08 12:45:26.223900 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBAutoScalingGroup --region us-east-1
2022/03/08 12:45:26.766000 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBBeanstalkMetadata --region us-east-1
2022/03/08 12:45:27.339055 [INFO] Generating rsyslog config from Procfile
2022/03/08 12:45:27.340167 [INFO] Running command /bin/sh -c systemctl restart rsyslog.service
2022/03/08 12:45:27.389097 [INFO] Executing instruction: PostBuildEbExtension
2022/03/08 12:45:27.389143 [INFO] Running command /bin/sh -c rbenv global
2022/03/08 12:45:27.493375 [INFO] 2.7.5

2022/03/08 12:45:27.493403 [INFO] Starting executing the config set Infra-EmbeddedPostBuild.
2022/03/08 12:45:27.493424 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:us-east-1:stack/awseb-stack/134436f0-a9af-11ea-8b63-0addedaaaadd -r AWSEBAutoScalingGroup --region us-east-1 --configsets Infra-EmbeddedPostBuild
2022/03/08 12:45:28.016424 [INFO] Finished executing the config set Infra-EmbeddedPostBuild.

2022/03/08 12:45:28.016450 [INFO] Executing instruction: CleanEbExtensions
2022/03/08 12:45:28.016487 [INFO] Cleaned ebextensions subdirectories from app staging directory.
2022/03/08 12:45:28.016492 [INFO] Executing instruction: RunAppDeployPreDeployHooks
2022/03/08 12:45:28.016510 [INFO] Executing platform hooks in .platform/hooks/predeploy/
2022/03/08 12:45:28.016545 [INFO] The dir .platform/hooks/predeploy/ does not exist
2022/03/08 12:45:28.016553 [INFO] Executing instruction: stop X-Ray
2022/03/08 12:45:28.016557 [INFO] stop X-Ray ...
2022/03/08 12:45:28.016571 [INFO] Running command /bin/sh -c systemctl show -p PartOf xray.service
2022/03/08 12:45:28.029596 [WARN] stopProcess Warning: process xray is not registered 
2022/03/08 12:45:28.029623 [INFO] Running command /bin/sh -c systemctl stop xray.service
2022/03/08 12:45:28.043064 [INFO] Executing instruction: stop proxy
2022/03/08 12:45:28.043086 [INFO] Running command /bin/sh -c systemctl show -p PartOf httpd.service
2022/03/08 12:45:28.054290 [WARN] deregisterProcess Warning: process httpd is not registered, skipping...

2022/03/08 12:45:28.054308 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/03/08 12:45:28.073921 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2022/03/08 12:45:28.083712 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/03/08 12:45:28.094777 [INFO] Running command /bin/sh -c systemctl stop nginx.service
2022/03/08 12:45:28.218708 [INFO] Running command /bin/sh -c systemctl disable nginx.service
2022/03/08 12:45:28.353598 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:45:28.453244 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:45:28.457902 [INFO] Executing instruction: flip ruby application
2022/03/08 12:45:28.457929 [INFO] Running command /bin/sh -c rbenv global
2022/03/08 12:45:28.471423 [INFO] 2.7.5

2022/03/08 12:45:28.471650 [INFO] Purge old process...
2022/03/08 12:45:28.471669 [INFO] Running command /bin/sh -c systemctl stop eb-app.target
2022/03/08 12:45:28.477198 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2022/03/08 12:45:28.482479 [INFO] web.service

2022/03/08 12:45:28.482497 [INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
2022/03/08 12:45:28.486493 [INFO] Running command /bin/sh -c systemctl is-active web.service
2022/03/08 12:45:28.489634 [INFO] Running command /bin/sh -c systemctl disable web.service
2022/03/08 12:45:28.553651 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:45:28.632539 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:45:29.497362 [INFO] Register application processes...
2022/03/08 12:45:29.497381 [INFO] Registering the proc: web

2022/03/08 12:45:29.497398 [INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
2022/03/08 12:45:29.519562 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:45:29.599337 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:45:29.602708 [INFO] Running command /bin/sh -c systemctl is-enabled eb-app.target
2022/03/08 12:45:29.606173 [INFO] Running command /bin/sh -c systemctl enable eb-app.target
2022/03/08 12:45:29.702370 [INFO] Running command /bin/sh -c systemctl start eb-app.target
2022/03/08 12:45:29.706730 [INFO] Running command /bin/sh -c systemctl enable web.service
2022/03/08 12:45:29.785888 [INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
2022/03/08 12:45:29.790300 [INFO] Running command /bin/sh -c systemctl is-active web.service
2022/03/08 12:45:29.793673 [INFO] Running command /bin/sh -c systemctl start web.service
2022/03/08 12:45:29.806026 [INFO] Executing instruction: start X-Ray
2022/03/08 12:45:29.806042 [INFO] X-Ray is not enabled.
2022/03/08 12:45:29.806047 [INFO] Executing instruction: start proxy with new configuration
2022/03/08 12:45:29.806064 [INFO] Running command /bin/sh -c /usr/sbin/nginx -t -c /var/proxy/staging/nginx/nginx.conf
2022/03/08 12:45:29.814352 [INFO] Running command /bin/sh -c cp -rp /var/proxy/staging/nginx/* /etc/nginx
2022/03/08 12:45:29.816566 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/03/08 12:45:29.825080 [INFO] Running command /bin/sh -c systemctl daemon-reload
2022/03/08 12:45:29.898123 [INFO] Running command /bin/sh -c systemctl reset-failed
2022/03/08 12:45:29.902509 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2022/03/08 12:45:29.908513 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2022/03/08 12:45:29.912131 [INFO] Running command /bin/sh -c systemctl start nginx.service
2022/03/08 12:45:29.989336 [INFO] Executing instruction: configureSqsd
2022/03/08 12:45:29.989353 [INFO] This is a web server environment instance, skip configure sqsd daemon ...
2022/03/08 12:45:29.989358 [INFO] Executing instruction: startSqsd
2022/03/08 12:45:29.989362 [INFO] This is a web server environment instance, skip start sqsd daemon ...
2022/03/08 12:45:29.989367 [INFO] Executing instruction: Track pids in healthd
2022/03/08 12:45:29.989372 [INFO] This is an enhanced health env...
2022/03/08 12:45:29.989388 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
2022/03/08 12:45:29.995412 [INFO] healthd.service cfn-hup.service nginx.service amazon-cloudwatch-agent.service

2022/03/08 12:45:29.995431 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2022/03/08 12:45:30.001598 [INFO] web.service

2022/03/08 12:45:30.001765 [INFO] Executing instruction: support rails logging
2022/03/08 12:45:30.001776 [INFO] adding builtin Rails logging support
2022/03/08 12:45:30.005448 [INFO] create soft link from /var/app/current/log/production.log to /var/app/containerfiles/logs/production.log
2022/03/08 12:45:30.005471 [INFO] Executing instruction: RunAppDeployPostDeployHooks
2022/03/08 12:45:30.005487 [INFO] Executing platform hooks in .platform/hooks/postdeploy/
2022/03/08 12:45:30.005510 [INFO] The dir .platform/hooks/postdeploy/ does not exist
2022/03/08 12:45:30.005516 [INFO] Executing cleanup logic
2022/03/08 12:45:30.005616 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment completed successfully.","timestamp":1646743530,"severity":"INFO"}]}]}

2022/03/08 12:45:30.005755 [INFO] Platform Engine finished execution on command: app-deploy

Here's what I have on .platform/hooks/prebuild/03_set_env_vars.sh

#!/bin/bash

# get the user to run the command as
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config platformconfig -k AppUser)

# export each key=value after escaping special chars
while read -r line; do
  key="$(cut -d= -f1 <<< "$line")"
  val="$(cut -d= -f2- <<< "$line" \
       | sed -r "/[\\$\"']/s/[\\$\"']/\\\\\\0/g")"
  export "$key=$val"
done < /opt/elasticbeanstalk/deployment/env

# run the command as the EB AppUser
eval "su -s /bin/bash -c \"$*\" \"$EB_APP_USER\""

as you can see the build finishes successfully. But below commands are failing:

[ec2-user]$ cd /var/app/current/
[ec2-user current]$ sudo su
[root current]# bundle exec rails console
Could not find awesome_print-1.9.2, factory_bot_rails-6.1.0, faker-2.17.0, letter_opener_web-1.4.0, listen-3.5.1, mocha-1.12.0, rspec-rails-4.0.2, rubocop-1.14.0, rubocop-rails-2.9.1, rubocop-rspec-2.3.0, httplog-1.4.3, capybara-3.35.3, selenium-webdriver-3.142.7, shoulda-matchers-4.5.1, rails-controller-testing-1.0.5, factory_bot-6.1.0, letter_opener-1.7.0, rspec-core-3.10.1, rspec-expectations-3.10.1, rspec-mocks-3.10.2, rspec-support-3.10.2, parallel-1.20.1, parser-3.0.1.1, rainbow-3.0.0, regexp_parser-2.1.1, rubocop-ast-1.5.0, ruby-progressbar-1.11.0, unicode-display_width-2.0.0, xpath-3.2.0, childprocess-3.0.0, launchy-2.5.0, diff-lcs-1.4.4, ast-2.4.2 in any of the sources

Above were the gems defined in the development and test block of Gemfile.

Then when I tried this:

[root current]# echo $AWS_STACK

[root current]#
nickpearson commented 2 years ago

The ebenv script needs to be used when calling other commands that need to access your Elastic Beanstalk environment properties as environment variables. You could do this with a .config file in .ebextensions, but here's how I do it:

Store my script in your app at bin/ebenv, and make sure it's executable. Then delete your 03_set_env_vars.sh file and replace it with a hook script at .platform/hooks/prebuild/03_ebenv.sh that copies the bin/ebenv file into place (to make it available in your PATH):

#!/bin/bash
set -xe
cp bin/ebenv /usr/local/bin/

Then run your commands like this:

sudo su -
cd /var/app/current
ebenv echo '$AWS_STACK'
ebenv bundle exec rails console

When used this way, ebenv will run the command you give it as the webapp user and with your environment properties made available as environment variables. (As I mentioned before, be sure to single-quote or escape shell variables so that they get expended inside the ebenv script and not before.)

I'm guessing the reason you got the "Could not find [gems]" error is because RACK_ENV is not defined in the shell environment, so Rails is defaulting to development, and those gems aren't installed in production. Using ebenv bundle exec rails console will correct that by adding RACK_ENV to the environment.

puneetpandey commented 2 years ago

Hi @nickpearson I followed your steps but it didn't work in my case. After deploy I am still not seeing any output of the command echo $RACK_ENV

I tried doing this also:

Under .platform/hooks/postdeploy/set_env_vars.sh, I entered below lines:

#!/bin/bash

# Setting up correct environment and ruby version so that bundle can load all gems
/opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "export \(.key)=\"\(.value)\""' > /etc/profile.d/sh.local
chmod u+x /etc/profile.d/sh.local
. /etc/profile.d/sh.local

and that also didn't work. But when I execute these lines directly on environment (after taking SSH), I can load rails console.

Phewww!! it's really a pain and I am sure there must be some silly mistake I am doing.

nickpearson commented 2 years ago

I'm glad you got it working and can start a Rails console.

For future reference, or for others who come across this in the future, note that echo $RACK_ENV won't work with my script; in that case, the command would be ebenv echo '$RACK_ENV' (see my previous comments for the explanation). It might also be worth noting that ebenv handles single-quote, double-quote, and dollar-sign characters in environment property values. I haven't tried the set_env_vars.sh way of doing this, so I don't know if it does or not, but just looking at the code, I suspect not. Lastly, set_env_vars.sh puts environment properties into the shell environment, whereas ebenv only makes the properties available to the command being executed.

puneetpandey commented 2 years ago

Thank you @nickpearson for all your help on this. I am also working with BUNDLER_DEPLOYMENT_MODE to see how it turns up