Weird getcwd error when deploying Ruby application on AL2

stefansundin commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments. They generate extra noise for issue followers and do not help prioritize the request.

Tell us about your request Hi. I was hoping you can help me figure out this weird issue that started after I migrated to Amazon Linux 2.

Is this request specific to an Elastic Beanstalk platform? Ruby 2.7 on Amazon Linux 2

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I recently migrated my app (it's open source) to Amazon Linux 2, and now every time I deploy, I see the following in my application logs:

May 10 17:27:56 ip-172-31-0-79 web: 13.59.9.120 - - [10/May/2020:17:27:56 +0000] "GET /metrics HTTP/1.1" 200 - 0.0561
May 10 17:28:21 ip-172-31-0-79 web: [16014] - Gracefully shutting down workers...
May 10 17:28:23 ip-172-31-0-79 web: shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
May 10 17:28:24 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)
May 10 17:28:24 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)
May 10 17:28:24 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)

# ... error repeated about 3000 times ...

May 10 17:29:29 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)
May 10 17:29:29 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)
May 10 17:29:29 ip-172-31-0-79 web: /opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle: No such file or directory - getcwd (Errno::ENOENT)
May 10 17:29:56 ip-172-31-0-79 web: 13.59.9.120 - - [10/May/2020:17:29:56 +0000] "GET /metrics HTTP/1.1" 200 - 0.0109

The error is also pushed to my exception reporting service (airbrake). Luckily I have modified it to not send duplicate errors, otherwise I would have seen 3000 errors on airbrake (and it would have consumed my quota pretty fast).

Airbrake does give me a backtrace which may be useful. Although it isn't enough for me to know where it comes from.

/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58 in getwd
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58 in pwd
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58 in block in pwd
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57 in synchronize
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57 in pwd
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253 in search_up
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:144 in print_major_deprecations!
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26 in ensure in start
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26 in start
/gems/bundler-2.1.2/exe/bundle:46 in block in <top (required)>
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123 in with_friendly_errors
/gems/bundler-2.1.2/exe/bundle:34 in <top (required)>
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle:23 in load
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle:23 in <main>
Caused by SignalException: SIGTERM
/gems/puma-4.3.3/lib/puma/cluster.rb:396 in block in setup_signals
/gems/puma-4.3.3/lib/puma/cluster.rb:491 in select
/gems/puma-4.3.3/lib/puma/cluster.rb:491 in run
/gems/puma-4.3.3/lib/puma/launcher.rb:172 in run
/gems/puma-4.3.3/lib/puma/cli.rb:80 in run
/gems/puma-4.3.3/bin/puma:10 in <top (required)>
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/puma:23 in load
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/puma:23 in <top (required)>
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63 in load
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63 in kernel_load
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28 in run
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476 in exec
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27 in run
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127 in invoke_command
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399 in dispatch
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30 in dispatch
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476 in start
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24 in start
/gems/bundler-2.1.2/exe/bundle:46 in block in <top (required)>
/opt/rubies/ruby-2.7.1/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123 in with_friendly_errors
/gems/bundler-2.1.2/exe/bundle:34 in <top (required)>
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle:23 in load
/opt/elasticbeanstalk/.rbenv/versions/2.7.1/bin/bundle:23 in <main>

My guess is that the old directory is deleted before the running process has a chance to stop?

Thanks!

Are you currently working around this issue? No workaround. It is not preventing my app from working. It is just a bit annoying and spams my logs.

Additional context You can deploy the app yourself if needed, I have tried to make it very easy. My app is hosted here: https://rssbox.us-west-2.elasticbeanstalk.com/

pprahlad commented 4 years ago

We recommend that members of the community ask for support, report bugs, etc. through community supported forums such as AWS Developer Forums or Stackoverflow.

Please continue to limit issues posted here to roadmap for Elastic Beanstalk. Thank you!

januszm commented 3 years ago

@stefansundin were you able to solve this issue? I'm wondering if your project uses gem sassc version 2.2 or newer. This gem's native extensions may take minutes to compile and perhaps aws has some kind of timeout on this - this is in relation to your comment:

My guess is that the old directory is deleted before the running process has a chance to stop?

Whole deployment may be restarted and /var/app/staging changed causing getcwd error before sassc compilation has a chance to finish

stefansundin commented 3 years ago

Hi @januszm.

I don't remember doing anything further after filing this issue. I do not use that gem.

januszm commented 3 years ago

@pprahlad is there a timeout set on the deployment process that would cause it to fail because bundle install takes too long and /var/app/staging is changed in the meantime, causing getcwd error? I mean, different than the Command Timeout which is set to 600 seconds, 10 minutes. Looks like the deployment restarts automatically after 2.5 minutes.

januszm commented 3 years ago

@stefansundin if your app was Ruby on Rails 6.x application then it used sassc (via https://github.com/sass/sassc-rails/blob/master/sassc-rails.gemspec) and if it was relatively new sassc version which requires native extensions, you may have had the same issue as I, deployment was restarted and directory changed before sassc had a chance to complete.

stefansundin commented 3 years ago

It is not a Rails app. If you want to look at my app then the source code is here: https://github.com/stefansundin/rssbox.

puneetpandey commented 3 years ago

Since I am also facing the similar issue, hence instead of opening a new ticket I am adding my issue as a comment here

I am facing similar issue when deploying my rails application on Amazon Linux 2.

My Gemfile:

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby '2.7.4'

gem 'actionpack-action_caching', '~> 1.2', '>= 1.2.2'
gem 'active_elastic_job'
gem 'active_model_serializers', '~> 0.10.10'
gem 'amazon_pay'
gem 'apipie-rails'
gem 'audited', '~> 4.9'
gem 'avatax'
gem 'aws-sdk-s3', '~> 1'
gem 'bootsnap', '~> 1.4.2'
gem 'bootstrap-filestyle-rails'
gem 'bootstrap-sass'
gem 'braintree'
gem 'browser'
gem 'chartjs-ror'
gem 'country-select'
gem 'dalli'
gem 'devise'
gem 'dropzonejs-rails'
gem 'elasticsearch-model', '~> 7.1'
gem 'elasticsearch-persistence', '~> 7.1'
gem 'elasticsearch-rails', '~> 7.1'
gem 'faraday_middleware'
gem 'flutie'
gem 'fog-aws'
gem 'font_assets'
gem 'friendly_id'
gem 'geocoder'
gem 'google-apis-analyticsreporting_v4'
#gem 'gridhook', git: 'https://github.com/GuillaumeWrobel/gridhook.git', branch: 'master'
gem 'hairtrigger'
gem 'haml', '~> 5.1', '>= 5.1.2'
gem 'inky-rb', require: 'inky'
gem 'jbuilder', '~> 2.10'
gem 'jhtmlarea'
gem 'jquery-datetimepicker-rails'
gem 'jquery-rails'
gem 'jquery-ui-rails', '5.0.3'
gem 'kaminari'
gem 'masonry-rails'
gem 'memcachier'
gem 'newrelic_rpm'
gem 'paperclip', '~> 6.0.0'
gem 'paperclip-compression'
gem 'pg'
gem 'pghero'
gem 'pg_search'
gem 'premailer-rails' # Stylesheet inlining for email
gem 'pry-rails'
gem 'puma', '~> 5.3.2'
gem 'puma_worker_killer'
gem 'rack', '~> 2.1', '>= 2.1.2'
gem 'rack-cors', require: 'rack/cors'
gem 'rack-maintenance'
gem 'rack-tracker'
gem 'rails', '~> 6.1', '>= 6.1.3.1'
gem 'rails-healthcheck'
gem 'recaptcha', require: 'recaptcha/rails'
gem 'ruby_parser', '~> 3.11'
gem 'rubyzip', '>= 1.0.0'
#gem 's3_direct_upload', git: 'https://github.com/waynehoover/s3_direct_upload.git', branch: 'master'
gem 's3_direct_upload', '~> 0.1.7'
gem 'sass-rails', '~> 5.0'
gem 'sdoc', '~> 0.4.0', group: :doc
gem 'sendgrid'
gem 'simple_form'
gem 'sitemap_generator'
gem "slack-notifier"
gem 'spreadsheet'
gem 'turbolinks', '~> 5.2.0'
gem 'uglifier', '>= 1.3.0'
gem 'view_component', require: 'view_component/engine'
gem 'wicked_pdf', '~> 2.0', '>= 2.0.1'
gem 'wkhtmltopdf-binary', '0.12.6'

group :develop, :stage, :production do
  gem 'honeybadger'
  gem 'rails_12factor'
end

group :development, :test do
  gem 'awesome_print'
  gem 'factory_bot_rails'
  gem 'faker'
  gem 'letter_opener_web'
  gem 'listen'
  gem 'mocha'
  gem 'rspec-rails', '~> 4.0.0.beta'
  gem 'rubocop', require: false
  gem 'rubocop-rails', require: false
  gem 'rubocop-rspec', require: false
end

group :development do
  gem 'httplog'
end

group :test do
  gem 'capybara'
  gem 'selenium-webdriver'
  gem 'shoulda-matchers'
  gem 'rails-controller-testing'
end

inside .ebextensions I have:

00_linux_packages.config

packages:
  yum:
    git: []
    patch: []
    ImageMagick: []
    ImageMagick-devel: []

01_rails_env.config

option_settings:
  aws:elasticbeanstalk:application:environment:
    BUNDLE_WITHOUT: test
    BUNDLER_DEPLOYMENT_MODE: true
    RAILS_SKIP_ASSET_COMPILATION: false
    RAILS_SKIP_MIGRATIONS: false
    LOGGING: debug
    BUNDLE_DISABLE_SHARED_GEMS: "1"
    BUNDLE_PATH: "vendor/bundle"

03_puma_conf.config

files:
  "/opt/elasticbeanstalk/config/private/pumaconf.rb":
    mode: "000644"
    owner: root
    group: root
    content: |
      directory '/var/app/current'
      # threads 8, 32
      threads 1, 1
      # workers %x(grep -c processor /proc/cpuinfo)
      workers 2 * Integer(%x(grep -c processor /proc/cpuinfo))
      bind 'unix:///var/run/puma/my_app.sock'
      pidfile '/var/run/puma/puma.pid'
      stdout_redirect '/var/log/puma/puma.log', '/var/log/puma/puma.log', true
      # daemonize false
      # preload
      preload_app!
      on_worker_boot do
        ActiveRecord::Base.connection_pool.disconnect!
        ActiveRecord::Base.establish_connection
      end

here's my Procfile: web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb
here's .platform/nginx/conf.d/proxy.conf

client_max_body_size 30M;
proxy_send_timeout 600;
proxy_read_timeout 1h;
send_timeout 600;

upstream backend {
  server unix:///var/run/puma/my_app.sock ;
}

server {
  listen 80;

  access_log /var/log/nginx/access.log;
  error_log /var/log/nginx/error.log;

  server_name *.cmgresearch.net;

  large_client_header_buffers 8 32k;

  location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_buffers 8 32k;
    proxy_buffer_size 64k;

    proxy_pass http://backend;
    proxy_redirect off;

    location /assets {
      root /var/app/current/public;
    }

    # enables WS support
    location /cable {
      proxy_pass http://backend;
      proxy_http_version 1.1;
      proxy_set_header Upgrade "websocket";
      proxy_set_header Connection "Upgrade";
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
  }
}

And below is the trail of error I am seeing (from /var/log/eb-engine.log):

[INFO] Executing instruction: StageApplication
[INFO] extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/
[INFO] Running command /bin/sh -c /usr/bin/unzip -q -o /opt/elasticbeanstalk/deployment/app_source_bundle -d /var/app/staging/
[INFO] finished extracting /opt/elasticbeanstalk/deployment/app_source_bundle to /var/app/staging/ successfully
[INFO] Executing instruction: RunAppDeployPreBuildHooks
[INFO] The dir .platform/hooks/prebuild/ does not exist in the application. Skipping this step...
[INFO] Executing instruction: stage ruby application
[INFO] stage ruby application ....
[INFO] Running command /bin/sh -c bundle config set --local deployment true
[INFO] Fetching gem metadata from https://rubygems.org/
Fetching gem metadata from https://rubygems.org/.......
Fetching rake 13.0.3
Installing rake 13.0.3
Fetching concurrent-ruby 1.1.8
Fetching minitest 5.14.4
Installing concurrent-ruby 1.1.8
Installing minitest 5.14.4
Fetching zeitwerk 2.4.2
Installing zeitwerk 2.4.2
Fetching builder 3.2.4
Fetching erubi 1.10.0
Installing builder 3.2.4
Installing erubi 1.10.0
Fetching mini_portile2 2.5.1
Fetching racc 1.5.2
Installing mini_portile2 2.5.1
Installing racc 1.5.2 with native extensions
Fetching crass 1.0.6
Installing crass 1.0.6
Fetching rack 2.2.3
Installing rack 2.2.3
Fetching nio4r 2.5.7
Installing nio4r 2.5.7 with native extensions
Fetching websocket-extensions 0.1.5
Installing websocket-extensions 0.1.5
Fetching marcel 1.0.1
Installing marcel 1.0.1
Fetching mini_mime 1.0.3
Installing mini_mime 1.0.3
Fetching aws-eventstream 1.1.1
Installing aws-eventstream 1.1.1
Fetching aws-partitions 1.453.0
Installing aws-partitions 1.453.0
Fetching jmespath 1.4.0
Installing jmespath 1.4.0
Using bundler 2.2.22
Fetching method_source 1.0.0
Installing method_source 1.0.0
Fetching thor 1.1.0
Installing thor 1.1.0
Fetching jsonapi-renderer 0.2.2
Installing jsonapi-renderer 0.2.2
Fetching public_suffix 4.0.6
Installing public_suffix 4.0.6
Fetching amazon_pay 2.6.1
Installing amazon_pay 2.6.1
Fetching ast 2.4.2
Installing ast 2.4.2
Fetching execjs 2.7.0
Installing execjs 2.7.0
Fetching faraday-excon 1.1.0
Installing faraday-excon 1.1.0
Fetching faraday-net_http 1.0.1
Installing faraday-net_http 1.0.1
Fetching faraday-net_http_persistent 1.1.0
Installing faraday-net_http_persistent 1.1.0
Fetching multipart-post 2.1.1
Installing multipart-post 2.1.1
Fetching ruby2_keywords 0.0.4
Installing ruby2_keywords 0.0.4
Fetching multi_json 1.15.0
Installing multi_json 1.15.0
Fetching awesome_print 1.9.2
Installing awesome_print 1.9.2
Fetching bcrypt 3.1.16
Installing bcrypt 3.1.16 with native extensions

[ERROR] An error occurred during execution of command [app-deploy] - [stage ruby application]. Stop running the command. Error: install dependencies in Gemfile failed with error Command /bin/sh -c bundle install failed with error exit status 1. Stderr:Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:240:in `find_file'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:232:in `find_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:47:in `in_bundle?'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/env.rb:31:in `report'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:100:in `request_issue_report_for'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:50:in `log_error'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:134:in `rescue in with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:126:in `with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:37:in `<top (required)>'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `load'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `<main>'
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:144:in `print_major_deprecations!'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26:in `ensure in start'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26:in `start'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:49:in `block in <top (required)>'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:128:in `with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:37:in `<top (required)>'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `load'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `<main>'
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:240:in `find_file'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:232:in `find_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:20:in `default_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:225:in `require_tree_for_spec'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:191:in `do_install'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:62:in `apply_func'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:57:in `block in process_queue'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:54:in `loop'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:54:in `process_queue'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:88:in `block (2 levels) in create_threads'

[INFO] Executing cleanup logic
[INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to install dependency gems that you defined in 'Gemfile'. For details, see 'eb-engine.log'. The deployment failed.","timestamp":1628163393,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1628163393,"severity":"ERROR"}]}]}

[INFO] Platform Engine finished execution on command: app-deploy

This can be related to the timeout issue but it looks to me it is not related to sassc gem.

Please suggest @januszm

januszm commented 3 years ago

You see getcwd: cannot access parent directories: No such file or directory because EB is trying to repeat the deployment process endlessly. I mean, it changes the /var/app/staging directory before it is able to finish the current action. getcwd fails because the directory was already replaced. SSH to the server eb ssh and check logs (with tail -1000 or cat)

/var/log/puma/puma.log
/var/log/eb-engine.log
/var/log/cfn-init.log
/var/log/nginx/error.log

And see if anything shows an error that is preventing the server from starting, building assets, migrating the database, or installing gems.

I recommend locking the sassc version in Gemfile to 2.1.0 (add the gem 'sassc', '2.1.0' line to Gemfile just after or before the gem 'sass-rails', '~> 5.0' and bundle it), this will speed up the process and debugging considerably. If timeout was the issue then this should work.

puneetpandey commented 3 years ago

Hi @januszm thank you for replying. For sassc, I am already using sassc (2.4.0) > 2.1.0 & 2.2. Do you suggest to use 2.1.0?

For the logs, I am checking all the mentioned places now

puneetpandey commented 3 years ago

Also this issue shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory does not appear everytime. For example, I tried terminating the EC2 instance, created those from fresh and re-deploy the application to both web and worker ec2 instances and this time Web build is passed but worker is still failing. However I still cannot open the web application because of unknown issue.

Below is the same error I received on eb-engine.log from worker's EC2 instance

[INFO] Fetching gem metadata from https://rubygems.org/
Fetching gem metadata from https://rubygems.org/.......
Fetching rake 13.0.3
Installing rake 13.0.3
Fetching minitest 5.14.4
Fetching concurrent-ruby 1.1.8
Installing minitest 5.14.4
Installing concurrent-ruby 1.1.8
Fetching zeitwerk 2.4.2
Installing zeitwerk 2.4.2
Fetching builder 3.2.4
Installing builder 3.2.4
Fetching erubi 1.10.0
Installing erubi 1.10.0
Fetching mini_portile2 2.5.1
Installing mini_portile2 2.5.1
Fetching racc 1.5.2
Fetching crass 1.0.6
Installing crass 1.0.6
Fetching rack 2.2.3
Installing racc 1.5.2 with native extensions
Installing rack 2.2.3
Fetching nio4r 2.5.7
Installing nio4r 2.5.7 with native extensions
Fetching websocket-extensions 0.1.5
Installing websocket-extensions 0.1.5
Fetching marcel 1.0.1
Installing marcel 1.0.1
Fetching mini_mime 1.0.3
Installing mini_mime 1.0.3
Fetching aws-eventstream 1.1.1
Installing aws-eventstream 1.1.1
Fetching aws-partitions 1.453.0
Installing aws-partitions 1.453.0
Fetching jmespath 1.4.0
Installing jmespath 1.4.0
Using bundler 2.2.22
Fetching method_source 1.0.0
Installing method_source 1.0.0
Fetching thor 1.1.0
Installing thor 1.1.0
Fetching jsonapi-renderer 0.2.2
Installing jsonapi-renderer 0.2.2
Fetching public_suffix 4.0.6
Installing public_suffix 4.0.6
Fetching amazon_pay 2.6.1
Installing amazon_pay 2.6.1
Fetching ast 2.4.2
Installing ast 2.4.2
Fetching execjs 2.7.0
Installing execjs 2.7.0
Fetching faraday-excon 1.1.0
Installing faraday-excon 1.1.0
Fetching faraday-net_http 1.0.1
Installing faraday-net_http 1.0.1
Fetching faraday-net_http_persistent 1.1.0
Installing faraday-net_http_persistent 1.1.0
Fetching multipart-post 2.1.1
Installing multipart-post 2.1.1
Fetching ruby2_keywords 0.0.4
Installing ruby2_keywords 0.0.4
Fetching multi_json 1.15.0
Installing multi_json 1.15.0
Fetching awesome_print 1.9.2
Installing awesome_print 1.9.2
Fetching bcrypt 3.1.16
Installing bcrypt 3.1.16 with native extensions
Fetching msgpack 1.4.2
Installing msgpack 1.4.2 with native extensions
Fetching bootstrap-filestyle-rails 1.2.1
Installing bootstrap-filestyle-rails 1.2.1
Fetching ffi 1.15.0
Installing ffi 1.15.0 with native extensions

[ERROR] An error occurred during execution of command [app-deploy] - [stage ruby application]. Stop running the command. Error: install dependencies in Gemfile failed with error Command /bin/sh -c bundle install failed with error exit status 1. Stderr:Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:240:in `find_file'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:232:in `find_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:47:in `in_bundle?'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/env.rb:31:in `report'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:100:in `request_issue_report_for'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:50:in `log_error'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:134:in `rescue in with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:126:in `with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:37:in `<top (required)>'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `load'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `<main>'
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:144:in `print_major_deprecations!'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26:in `ensure in start'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:26:in `start'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:49:in `block in <top (required)>'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:128:in `with_friendly_errors'
    from /opt/rubies/ruby-2.7.4/lib/ruby/gems/2.7.0/gems/bundler-2.2.22/exe/bundle:37:in `<top (required)>'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `load'
    from /opt/elasticbeanstalk/.rbenv/versions/2.7.4/bin/bundle:23:in `<main>'
/opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `getwd': No such file or directory - getcwd (Errno::ENOENT)
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:58:in `block in pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `synchronize'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:57:in `pwd'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:253:in `search_up'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:240:in `find_file'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:232:in `find_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/shared_helpers.rb:20:in `default_gemfile'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:225:in `require_tree_for_spec'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:191:in `do_install'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:62:in `apply_func'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:57:in `block in process_queue'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:54:in `loop'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:54:in `process_queue'
    from /opt/rubies/ruby-2.7.4/lib/ruby/site_ruby/2.7.0/bundler/worker.rb:88:in `block (2 levels) in create_threads'

but the web build passed:

[INFO] Fetching gem metadata from https://rubygems.org/
Fetching gem metadata from https://rubygems.org/.......
Fetching rake 13.0.3
Installing rake 13.0.3
Fetching concurrent-ruby 1.1.8
Fetching minitest 5.14.4
Installing minitest 5.14.4
Installing concurrent-ruby 1.1.8
Fetching zeitwerk 2.4.2
Installing zeitwerk 2.4.2
Fetching builder 3.2.4
Installing builder 3.2.4
Fetching erubi 1.10.0
Installing erubi 1.10.0
Fetching mini_portile2 2.5.1
Fetching racc 1.5.2
Installing mini_portile2 2.5.1
Fetching crass 1.0.6
Installing racc 1.5.2 with native extensions
Installing crass 1.0.6
Fetching rack 2.2.3
Installing rack 2.2.3
Fetching nio4r 2.5.7
Installing nio4r 2.5.7 with native extensions
Fetching websocket-extensions 0.1.5
Installing websocket-extensions 0.1.5
Fetching marcel 1.0.1
Installing marcel 1.0.1
Fetching mini_mime 1.0.3
Installing mini_mime 1.0.3
Fetching aws-eventstream 1.1.1
Installing aws-eventstream 1.1.1
Fetching aws-partitions 1.453.0
Installing aws-partitions 1.453.0
Fetching jmespath 1.4.0
Installing jmespath 1.4.0
Using bundler 2.2.22
Fetching method_source 1.0.0
Installing method_source 1.0.0
Fetching thor 1.1.0
Installing thor 1.1.0
Fetching jsonapi-renderer 0.2.2
Installing jsonapi-renderer 0.2.2
Fetching public_suffix 4.0.6
Installing public_suffix 4.0.6
Fetching amazon_pay 2.6.1
Installing amazon_pay 2.6.1
Fetching ast 2.4.2
Installing ast 2.4.2
Fetching execjs 2.7.0
Installing execjs 2.7.0
Fetching faraday-excon 1.1.0
Installing faraday-excon 1.1.0
Fetching faraday-net_http 1.0.1
Installing faraday-net_http 1.0.1
Fetching faraday-net_http_persistent 1.1.0
Installing faraday-net_http_persistent 1.1.0
Fetching multipart-post 2.1.1
Installing multipart-post 2.1.1
Fetching ruby2_keywords 0.0.4
Installing ruby2_keywords 0.0.4
Fetching multi_json 1.15.0
Installing multi_json 1.15.0
Fetching awesome_print 1.9.2
Installing awesome_print 1.9.2
Fetching bcrypt 3.1.16
Installing bcrypt 3.1.16 with native extensions
Fetching msgpack 1.4.2
Installing msgpack 1.4.2 with native extensions
Fetching bootstrap-filestyle-rails 1.2.1
Installing bootstrap-filestyle-rails 1.2.1
Fetching ffi 1.15.0
Installing ffi 1.15.0 with native extensions
Fetching libxml-ruby 3.2.1
Installing libxml-ruby 3.2.1 with native extensions
Fetching browser 5.3.1
Installing browser 5.3.1
Fetching climate_control 0.2.0
Installing climate_control 0.2.0
Fetching coderay 1.1.3
Installing coderay 1.1.3
Fetching coffee-script-source 1.12.2
Installing coffee-script-source 1.12.2
Fetching country-select 1.2.1
Installing country-select 1.2.1
Fetching dalli 2.7.11
Installing dalli 2.7.11
Fetching declarative 0.0.20
Installing declarative 0.0.20
Fetching orm_adapter 0.5.0
Installing orm_adapter 0.5.0
Fetching diff-lcs 1.4.4
Installing diff-lcs 1.4.4
Fetching hashie 4.1.0
Installing hashie 4.1.0
Fetching elasticsearch-rails 7.1.1
Installing elasticsearch-rails 7.1.1
Fetching excon 0.81.0
Installing excon 0.81.0
Fetching flutie 2.2.0
Installing flutie 2.2.0
Fetching formatador 0.2.5
Installing formatador 0.2.5
Fetching mime-types-data 3.2021.0225
Installing mime-types-data 3.2021.0225
Fetching ipaddress 0.8.3
Installing ipaddress 0.8.3
Fetching foundation_emails 2.2.1.0
Installing foundation_emails 2.2.1.0
Fetching geocoder 1.6.7
Installing geocoder 1.6.7
Fetching jwt 2.2.3
Installing jwt 2.2.3
Fetching memoist 0.16.2
Installing memoist 0.16.2
Fetching os 1.0.1
Installing os 1.0.1
Fetching httpclient 2.8.3
Installing httpclient 2.8.3
Fetching trailblazer-option 0.1.1
Installing trailblazer-option 0.1.1
Fetching uber 0.1.0
Installing uber 0.1.0
Fetching retriable 3.1.2
Installing retriable 3.1.2
Fetching rexml 3.2.5
Installing rexml 3.2.5
Fetching webrick 1.7.0
Installing webrick 1.7.0
Fetching sexp_processor 4.15.2
Installing sexp_processor 4.15.2
Fetching temple 0.8.2
Installing temple 0.8.2
Fetching tilt 2.0.10
Installing tilt 2.0.10
Fetching honeybadger 4.8.0
Installing honeybadger 4.8.0
Fetching htmlentities 4.3.4
Installing htmlentities 4.3.4
Fetching rainbow 3.0.0
Installing rainbow 3.0.0
Fetching jquery-datetimepicker-rails 2.4.1.0
Installing jquery-datetimepicker-rails 2.4.1.0
Fetching rb-fsevent 0.10.4
Installing rb-fsevent 0.10.4
Fetching json 1.8.6
Installing json 1.8.6 with native extensions
Fetching kaminari-core 1.2.1
Installing kaminari-core 1.2.1
Fetching memcachier 0.0.2
Installing memcachier 0.0.2
Fetching mocha 1.12.0
Installing mocha 1.12.0
Fetching newrelic_rpm 7.0.0
Installing newrelic_rpm 7.0.0
Fetching parallel 1.20.1
Installing parallel 1.20.1
Fetching pg 1.2.3
Installing pg 1.2.3 with native extensions
Fetching rails_serve_static_assets 0.0.5
Installing rails_serve_static_assets 0.0.5
Fetching rails_stdout_logging 0.0.5
Installing rails_stdout_logging 0.0.5
Fetching rdoc 4.3.0
Installing rdoc 4.3.0
Fetching regexp_parser 2.1.1
Installing regexp_parser 2.1.1
Fetching rspec-support 3.10.2
Installing rspec-support 3.10.2
Fetching ruby-progressbar 1.11.0
Installing ruby-progressbar 1.11.0
Fetching unicode-display_width 2.0.0
Installing unicode-display_width 2.0.0
Fetching ruby-ole 1.2.12.2
Installing ruby-ole 1.2.12.2
Fetching rubyzip 2.3.0
Installing rubyzip 2.3.0
Fetching slack-notifier 2.4.0
Installing slack-notifier 2.4.0
Fetching turbolinks-source 5.2.0
Installing turbolinks-source 5.2.0
Fetching wkhtmltopdf-binary 0.12.6
Fetching sitemap_generator 6.1.2
Installing sitemap_generator 6.1.2
Fetching i18n 1.8.10
Installing i18n 1.8.10
Fetching tzinfo 2.0.4
Installing tzinfo 2.0.4
Fetching rack-test 1.1.0
Installing rack-test 1.1.0
Fetching sprockets 3.7.2
Installing sprockets 3.7.2
Fetching warden 1.2.9
Installing warden 1.2.9
Fetching font_assets 0.1.14
Installing font_assets 0.1.14
Fetching rack-cors 1.1.1
Installing rack-cors 1.1.1
Fetching rack-maintenance 2.1.0
Installing rack-maintenance 2.1.0
Fetching nokogiri 1.11.3 (x86_64-linux)
Installing nokogiri 1.11.3 (x86_64-linux)
Fetching websocket-driver 0.7.3
Installing websocket-driver 0.7.3 with native extensions
Fetching mail 2.7.1
Installing mail 2.7.1
Fetching aws-sigv4 1.2.3
Installing aws-sigv4 1.2.3
Fetching addressable 2.7.0
Installing addressable 2.7.0
Fetching parser 3.0.1.1
Installing parser 3.0.1.1
Fetching autoprefixer-rails 10.2.4.0
Installing autoprefixer-rails 10.2.4.0
Fetching uglifier 4.2.0
Installing uglifier 4.2.0
Fetching faraday 1.4.1
Installing faraday 1.4.1
Fetching elasticsearch-api 7.12.0
Installing elasticsearch-api 7.12.0
Fetching puma 5.3.2
Installing puma 5.3.2 with native extensions
Installing wkhtmltopdf-binary 0.12.6
Fetching bootsnap 1.4.9
Installing bootsnap 1.4.9 with native extensions
Fetching braintree 3.4.0
Installing braintree 3.4.0
Fetching terrapin 0.6.0
Installing terrapin 0.6.0
Fetching pry 0.14.1
Installing pry 0.14.1
Fetching coffee-script 2.4.1
Installing coffee-script 2.4.1
Fetching mime-types 3.3.1
Installing mime-types 3.3.1
Fetching representable 3.1.1
Installing representable 3.1.1
Fetching ruby_parser 3.15.1
Installing ruby_parser 3.15.1
Fetching haml 5.2.1
Installing haml 5.2.1
Fetching httplog 1.4.3
Installing httplog 1.4.3
Fetching recaptcha 5.7.0
Installing recaptcha 5.7.0
Fetching sendgrid 1.2.4
Installing sendgrid 1.2.4
Fetching sassc 2.4.0
Installing sassc 2.4.0 with native extensions
Fetching get_process_mem 0.2.7
Installing get_process_mem 0.2.7
Fetching rb-inotify 0.10.1
Installing rb-inotify 0.10.1
Fetching rails_12factor 0.0.3
Installing rails_12factor 0.0.3
Fetching sdoc 0.4.2
Installing sdoc 0.4.2
Fetching rspec-core 3.10.1
Installing rspec-core 3.10.1
Fetching rspec-expectations 3.10.1
Installing rspec-expectations 3.10.1
Fetching rspec-mocks 3.10.2
Installing rspec-mocks 3.10.2
Fetching spreadsheet 1.2.8
Installing spreadsheet 1.2.8
Fetching turbolinks 5.2.1
Installing turbolinks 5.2.1
Fetching faker 2.17.0
Installing faker 2.17.0
Fetching activesupport 6.1.3.2
Installing activesupport 6.1.3.2
Fetching loofah 2.9.1
Installing loofah 2.9.1
Fetching inky-rb 1.3.8.0
Installing inky-rb 1.3.8.0
Fetching mimemagic 0.3.10
Installing mimemagic 0.3.10 with native extensions
Fetching aws-sdk-core 3.114.0
Installing aws-sdk-core 3.114.0
Fetching css_parser 1.9.0
Installing css_parser 1.9.0
Fetching launchy 2.5.0
Installing launchy 2.5.0
Fetching rubocop-ast 1.5.0
Installing rubocop-ast 1.5.0
Fetching faraday_middleware 1.0.0
Installing faraday_middleware 1.0.0
Fetching elasticsearch-transport 7.12.0
Installing elasticsearch-transport 7.12.0
Fetching signet 0.15.0
Installing signet 0.15.0
Fetching pry-rails 0.3.9
Installing pry-rails 0.3.9
Fetching fog-core 2.2.4
Installing fog-core 2.2.4
Fetching ruby2ruby 2.4.4
Installing ruby2ruby 2.4.4
Fetching puma_worker_killer 0.3.1
Installing puma_worker_killer 0.3.1
Fetching sass-listen 4.0.0
Installing sass-listen 4.0.0
Fetching listen 3.5.1
Installing listen 3.5.1
Fetching rails-dom-testing 2.0.3
Installing rails-dom-testing 2.0.3
Fetching globalid 0.4.2
Installing globalid 0.4.2
Fetching activemodel 6.1.3.2
Installing activemodel 6.1.3.2
Fetching case_transform 0.2
Installing case_transform 0.2
Fetching factory_bot 6.1.0
Installing factory_bot 6.1.0
Fetching jbuilder 2.11.2
Installing jbuilder 2.11.2
Fetching rack-tracker 1.12.1
Installing rack-tracker 1.12.1
Fetching view_component 2.31.1
Installing view_component 2.31.1
Fetching wicked_pdf 2.1.0
Installing wicked_pdf 2.1.0
Fetching rails-html-sanitizer 1.3.0
Installing rails-html-sanitizer 1.3.0
Fetching aws-sdk-sqs 1.38.0
Installing aws-sdk-sqs 1.38.0
Fetching aws-sdk-kms 1.43.0
Installing aws-sdk-kms 1.43.0
Fetching premailer 1.14.3
Installing premailer 1.14.3
Fetching letter_opener 1.7.0
Installing letter_opener 1.7.0
Fetching rubocop 1.14.0
Installing rubocop 1.14.0
Fetching avatax 21.3.1
Installing avatax 21.3.1
Fetching elasticsearch 7.12.0
Installing elasticsearch 7.12.0
Fetching googleauth 0.16.2
Installing googleauth 0.16.2
Fetching fog-json 1.2.0
Installing fog-json 1.2.0
Fetching fog-xml 0.1.3
Installing fog-xml 0.1.3
Fetching sass 3.7.4
Installing sass 3.7.4
Fetching activejob 6.1.3.2
Installing activejob 6.1.3.2
Fetching activerecord 6.1.3.2
Installing activerecord 6.1.3.2
Fetching paperclip 6.0.0
Installing paperclip 6.0.0
Fetching actionview 6.1.3.2
Installing actionview 6.1.3.2
Fetching aws-sdk-s3 1.94.1
Installing aws-sdk-s3 1.94.1
Fetching rubocop-rails 2.9.1
Installing rubocop-rails 2.9.1
Fetching rubocop-rspec 2.3.0
Installing rubocop-rspec 2.3.0
Fetching elasticsearch-model 7.1.1
Installing elasticsearch-model 7.1.1
Fetching google-apis-core 0.3.0
Installing google-apis-core 0.3.0
Fetching fog-aws 3.10.0
Installing fog-aws 3.10.0
Fetching audited 4.10.0
Installing audited 4.10.0
Fetching friendly_id 5.4.2
Installing friendly_id 5.4.2
Fetching hairtrigger 0.2.24
Installing hairtrigger 0.2.24
Fetching kaminari-activerecord 1.2.1
Installing kaminari-activerecord 1.2.1
Fetching pg_search 2.3.5
Installing pg_search 2.3.5
Fetching pghero 2.8.1
Installing pghero 2.8.1
Fetching paperclip-compression 1.1.1
Installing paperclip-compression 1.1.1
Fetching actionpack 6.1.3.2
Installing actionpack 6.1.3.2
Fetching kaminari-actionview 1.2.1
Installing kaminari-actionview 1.2.1
Fetching elasticsearch-persistence 7.1.1
Installing elasticsearch-persistence 7.1.1
Fetching google-apis-analyticsreporting_v4 0.2.0
Installing google-apis-analyticsreporting_v4 0.2.0
Fetching actioncable 6.1.3.2
Installing actioncable 6.1.3.2
Fetching activestorage 6.1.3.2
Installing activestorage 6.1.3.2
Fetching actionmailer 6.1.3.2
Installing actionmailer 6.1.3.2
Fetching actionpack-action_caching 1.2.2
Installing actionpack-action_caching 1.2.2
Fetching railties 6.1.3.2
Installing railties 6.1.3.2
Fetching sprockets-rails 3.2.2
Installing sprockets-rails 3.2.2
Fetching active_model_serializers 0.10.12
Installing active_model_serializers 0.10.12
Fetching simple_form 5.1.0
Installing simple_form 5.1.0
Fetching kaminari 1.2.1
Installing kaminari 1.2.1
Fetching actionmailbox 6.1.3.2
Installing actionmailbox 6.1.3.2
Fetching actiontext 6.1.3.2
Installing actiontext 6.1.3.2
Fetching premailer-rails 1.11.1
Installing premailer-rails 1.11.1
Fetching coffee-rails 5.0.0
Installing coffee-rails 5.0.0
Fetching responders 3.0.1
Installing responders 3.0.1
Fetching factory_bot_rails 6.1.0
Installing factory_bot_rails 6.1.0
Fetching jquery-rails 4.4.0
Installing jquery-rails 4.4.0
Fetching jquery-fileupload-rails 0.4.7
Installing jquery-fileupload-rails 0.4.7
Fetching jquery-ui-rails 5.0.3
Installing jquery-ui-rails 5.0.3
Fetching letter_opener_web 1.4.0
Installing letter_opener_web 1.4.0
Fetching masonry-rails 0.2.4
Installing masonry-rails 0.2.4
Fetching rails-healthcheck 1.3.0
Installing rails-healthcheck 1.3.0
Fetching rspec-rails 4.0.2
Installing rspec-rails 4.0.2
Fetching sass-rails 5.1.0
Installing sass-rails 5.1.0
Fetching rails 6.1.3.2
Installing rails 6.1.3.2
Fetching devise 4.8.0
Installing devise 4.8.0
Fetching jhtmlarea 0.1.0
Installing jhtmlarea 0.1.0
Fetching active_elastic_job 3.0.0
Installing active_elastic_job 3.0.0
Fetching apipie-rails 0.5.18
Installing apipie-rails 0.5.18
Fetching chartjs-ror 3.6.4
Installing chartjs-ror 3.6.4
Fetching dropzonejs-rails 0.8.5
Installing dropzonejs-rails 0.8.5
Fetching s3_direct_upload 0.1.7
Installing s3_direct_upload 0.1.7
Fetching bootstrap-sass 3.4.1
Installing bootstrap-sass 3.4.1
Bundle complete! 88 Gemfile dependencies, 225 gems now installed.
Gems in the group 'test' were not installed.
Bundled gems are installed into `./vendor/bundle`
Post-install message from avatax:
********************************************************************************

AvaTax REST API
------------------------------
Our developer site documents the AvaTax REST API.
(http://developer.avatax.com).
Blog
----------------------------
The Developer Blog is a great place to learn more about the API and AvaTax integrations
Subscribe to the RSS feed be notified of new posts:
(http://developer.avatax.com/blog).

********************************************************************************
Post-install message from sass:

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

Post-install message from paperclip:
##################################################
#  NOTE FOR UPGRADING FROM 4.3.0 OR EARLIER      #
##################################################

Paperclip is now compatible with aws-sdk >= 2.0.0.

If you are using S3 storage, aws-sdk >= 2.0.0 requires you to make a few small
changes:

* You must set the `s3_region`
* If you are explicitly setting permissions anywhere, such as in an initializer,
  note that the format of the permissions changed from using an underscore to
  using a hyphen. For example, `:public_read` needs to be changed to
  `public-read`.

For a walkthrough of upgrading from 4 to 5 and aws-sdk >= 2.0 you can watch
http://rubythursday.com/episodes/ruby-snack-27-upgrade-paperclip-and-aws-sdk-in-prep-for-rails-5
Post-install message from rack-maintenance:

******** Rack::Maintenance 2.0.0 *********

Rack::Maintenance 2.0.0 has dropped the built-in filter for /assets.
This means that all your paths will now be blocked in maintenance
mode, unless you exclude them in the configuration.

To bring back pre-2.0.0 behaviour, configure the gem like so:

config.middleware.use 'Rack::Maintenance',
  file: Rails.root.join("public", "maintenance.html"),
  without: /\A\/assets/

[INFO] Running command /bin/sh -c rbenv rehash
[INFO] encountered a Gemfile.lock, setting proper permissions to user webapp
[INFO] Executing instruction: check default web server
[INFO] Executing instruction: check procfile
[INFO] detected Procfile in application source bundle ...
[INFO] Executing instruction: configure X-Ray
[INFO] X-Ray is not enabled.
[INFO] Executing instruction: Configure proxy Nginx
[INFO] configuring proxy...
[INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/webapp.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/webapp.conf
[INFO] Copying file /opt/elasticbeanstalk/config/private/nginx/elasticbeanstalk-nginx-ruby-upstream.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk-nginx-ruby-upstream.conf
[INFO] Running command /bin/sh -c cp -rp /var/app/staging/.platform/nginx/. /var/proxy/staging/nginx
[INFO] Executing instruction: configure healthd specific proxy conf
[INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
[WARN] Warning: process healthd is already registered...
Deregistering the process ...
[INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
[INFO] Running command /bin/sh -c systemctl is-active healthd.service
[INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
[INFO] Running command /bin/sh -c systemctl stop healthd.service
[INFO] Running command /bin/sh -c systemctl disable healthd.service
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Running command /bin/sh -c systemctl is-enabled aws-eb.target
[INFO] Running command /bin/sh -c systemctl enable aws-eb.target
[INFO] Running command /bin/sh -c systemctl start aws-eb.target
[INFO] Running command /bin/sh -c systemctl enable healthd.service
[INFO] Running command /bin/sh -c systemctl show -p PartOf healthd.service
[INFO] Running command /bin/sh -c systemctl is-active healthd.service
[INFO] Running command /bin/sh -c systemctl start healthd.service
[INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_logformat.conf to /var/proxy/staging/nginx/conf.d/healthd_logformat.conf
[INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd_nginx.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/healthd.conf
[INFO] Executing instruction: rake tasks
[INFO] Running command /bin/su webapp -c bundle exec /opt/elasticbeanstalk/config/private/checkforraketask.rb assets:precompile
[INFO] Found assets:precompile task in Rakefile
[INFO] Running command /bin/su webapp -c bundle exec rake assets:precompile
[INFO] ** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.8.0 framework=rails level=1 pid=7419
[INFO] Running command /bin/su webapp -c bundle exec /opt/elasticbeanstalk/config/private/checkforraketask.rb db:migrate
[INFO] Found db:migrate task in Rakefile
[INFO] Running command /bin/su webapp -c bundle exec rake db:migrate
[INFO] Executing instruction: configure log streaming
[INFO] start to get cloudwatch log client
[INFO] start to config log streaming
[INFO] start to configure log streaming config file
[INFO] start to create cloudwatch log stream
[INFO] start to create log streaming
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/eb-engine.log already exists. No need to create it.
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/eb-hooks.log already exists. No need to create it.
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/nginx/access.log already exists. No need to create it.
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/nginx/error.log already exists. No need to create it.
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/web.stdout.log already exists. No need to create it.
[WARN] Log stream for log group /aws/elasticbeanstalk/developWeb/var/log/puma/puma.log already exists. No need to create it.
[INFO] start to fetch config
[INFO] Running command /bin/sh -c /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -c file:/opt/aws/amazon-cloudwatch-agent/etc/beanstalk.json -s
[INFO] ****** processing amazon-cloudwatch-agent ******
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_beanstalk.json.tmp
Start configuration validation...
Valid Json input schema.
I! Detecting run_as_user...
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
Configuration validation second phase succeeded
Configuration validation succeeded
[INFO] Executing instruction: GetToggleForceRotate
[INFO] Checking if logs need forced rotation
[INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-e-ubgdnyfmhd-stack/STACK_ID -r AWSEBAutoScalingGroup --region us-east-1
[INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:stack/awseb-e-ubgdnyfmhd-stack/STACK_ID-r AWSEBBeanstalkMetadata --region us-east-1
[INFO] Generating rsyslog config from Procfile
[INFO] Running command /bin/sh -c systemctl restart rsyslog.service
[INFO] Executing instruction: PostBuildEbExtension
[INFO] Starting executing the config set Infra-EmbeddedPostBuild.
[INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:us-east-1:stack/awseb-e-ubgdnyfmhd-stack/STACK_ID -r AWSEBAutoScalingGroup --region us-east-1 --configsets Infra-EmbeddedPostBuild
[INFO] Finished executing the config set Infra-EmbeddedPostBuild.
[INFO] Executing instruction: CleanEbExtensions
[INFO] Cleaned ebextensions subdirectories from app staging directory.
[INFO] Executing instruction: RunAppDeployPreDeployHooks
[INFO] The dir .platform/hooks/predeploy/ does not exist in the application. Skipping this step...
[INFO] Executing instruction: stop X-Ray
[INFO] stop X-Ray ...
[INFO] Running command /bin/sh -c systemctl show -p PartOf xray.service
[WARN] stopProcess Warning: process xray is not registered 
[INFO] Running command /bin/sh -c systemctl stop xray.service
[INFO] Executing instruction: stop proxy
[INFO] Running command /bin/sh -c systemctl show -p PartOf httpd.service
[WARN] deregisterProcess Warning: process httpd is not registered, skipping...
[INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
[INFO] Running command /bin/sh -c systemctl is-active nginx.service
[INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
[INFO] Running command /bin/sh -c systemctl stop nginx.service
[INFO] Running command /bin/sh -c systemctl disable nginx.service
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Executing instruction: flip ruby application
[INFO] Purge old process...
[INFO] Running command /bin/sh -c systemctl stop eb-app.target
[INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
[INFO] web.service
[INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
[INFO] Running command /bin/sh -c systemctl is-active web.service
[INFO] Running command /bin/sh -c systemctl disable web.service
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Register application processes...
[INFO] Registering the proc: web
[INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Running command /bin/sh -c systemctl is-enabled eb-app.target
[INFO] Running command /bin/sh -c systemctl enable eb-app.target
[INFO] Running command /bin/sh -c systemctl start eb-app.target
[INFO] Running command /bin/sh -c systemctl enable web.service
[INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
[INFO] Running command /bin/sh -c systemctl is-active web.service
[INFO] Running command /bin/sh -c systemctl start web.service
[INFO] Executing instruction: start X-Ray
[INFO] X-Ray is not enabled.
[INFO] Executing instruction: start proxy with new configuration
[INFO] Running command /bin/sh -c /usr/sbin/nginx -t -c /var/proxy/staging/nginx/nginx.conf
[INFO] Running command /bin/sh -c cp -rp /var/proxy/staging/nginx/* /etc/nginx
[INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
[INFO] Running command /bin/sh -c systemctl daemon-reload
[INFO] Running command /bin/sh -c systemctl reset-failed
[INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
[INFO] Running command /bin/sh -c systemctl is-active nginx.service
[INFO] Running command /bin/sh -c systemctl start nginx.service
[INFO] Executing instruction: configureSqsd
[INFO] This is a web server environment instance, skip configure sqsd daemon ...
[INFO] Executing instruction: startSqsd
[INFO] This is a web server environment instance, skip start sqsd daemon ...
[INFO] Executing instruction: Track pids in healthd
[INFO] This is an enhanced health env...
[INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
[INFO] healthd.service amazon-cloudwatch-agent.service cfn-hup.service nginx.service
[INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
[INFO] web.service
[INFO] Executing instruction: support rails logging
[INFO] adding builtin Rails logging support
[INFO] create soft link from /var/app/current/log/develop.log to /var/app/containerfiles/logs/develop.log
[INFO] Executing instruction: RunAppDeployPostDeployHooks
[INFO] The dir .platform/hooks/postdeploy/ does not exist in the application. Skipping this step...
[INFO] Executing cleanup logic
[INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[{"msg":"Instance deployment completed successfully.","timestamp":1628247321,"severity":"INFO"}]}]}
[INFO] Platform Engine finished execution on command: app-deploy

Yet server is not reachable

Also checked that puma is running:

$ ps -ef | grep puma
healthd  11015     1  0 12:45 ?        00:00:02 puma 5.3.2 (tcp://127.0.0.1:22221) [healthd]
webapp   14934     1 96 13:01 ?        00:00:06 puma 5.3.2 (unix:///var/run/puma/my_app.sock) [current]
ec2-user 14973 14344  0 13:02 pts/0    00:00:00 grep --color=auto puma

puma.log missing at: /var/log/puma/puma/

Tried running / accessing console (via sudo) from /var/app/current

# bundle exec rails console
bundler: command not found: rails
Install missing gem executables with `bundle install`

These are multiple issues and I think some common configuration/setting will fix all/most of it.

Can you please suggest @januszm

puneetpandey commented 2 years ago

After a lot of struggle for months on this, I was finally able to migrate to AL2.

Few pointers that helped me:

I was trying to migrate from Ruby 2.6.8 to 2.7.5 - That didn't work. So I decided migrated to 2.6.9 and then to 2.7.5 - Though I am really not sure this helped me or not but it worked with other steps I followed.
My Application was using custom puma configuration and the version of Puma was different than the supported one, so the default configuration of Puma provided by AL2 didn't work and thus puma was unable to start. So I had to create Procfile and wrote a predeploy hook to make it work.
Same goes with nginx configuration -> I had to move all the configuration files from .ebextensions to .platform/nginx folder
My Application was unable to read the environmental variables - even after the successful deploy - so I had to wrote a postdeploy hook to load all ENV variables in shell.

I am still trying to figure out the way to fasten the deployment time as bundle install --local and rake assets:precompile both takes around 3-4 minutes each.

aws / elastic-beanstalk-roadmap

Weird getcwd error when deploying Ruby application on AL2 #52

Community Note