search

heroku / heroku-buildpack-ruby

Heroku's buildpack for Ruby applications.
MIT License
787 stars 4 forks source link

CI doesn't run all tests #1505

Open schneems opened 1 week ago

schneems commented 1 week ago

Problem described here https://github.com/heroku/heroku-buildpack-ruby/pull/1503#discussion_r1826067195

schneems commented 1 week ago 
$ bundle exec rspec-queue --dry-run --queue redis://localhost:6379 --build mybuild --worker myworker
Run options: include {:focused=>true}

All examples were filtered out; ignoring {:focused=>true}
......................................................................................................................

Finished in 0.0625 seconds (files took 0.28401 seconds to load)
118 examples, 0 failures

⛄️ 3.1.6 🚀 /Users/rschneeman/Documents/projects/work/heroku-buildpack-ruby (jwl/update-nodejs-default)
$ cat log/test_order.log
./spec/helpers/rails_runner_spec.rb[1:1]
./spec/hatchet/rails4_spec.rb[1:3]
./spec/helpers/ruby_version_spec.rb[1:4]
./spec/rake/deploy_check_spec.rb[1:2:5]
./spec/hatchet/rails7_spec.rb[1:2]
./spec/helpers/rake_runner_spec.rb[1:1]
./spec/helpers/binstub_check_spec.rb[1:2]
./spec/helpers/outdated_ruby_version_spec.rb[1:3]
./spec/hatchet/ci_spec.rb[1:4]
./spec/helpers/download_presence_spec.rb[1:2]
./spec/hatchet/bundler_spec.rb[1:1]
./spec/helpers/bundler_wrapper_spec.rb[4:3:2]
./spec/hatchet/rubies_spec.rb[2:1]
./spec/hatchet/rails23_spec.rb[1:1]
./spec/helpers/rake_runner_spec.rb[1:3]
./spec/hatchet/rails6_spec.rb[1:1]
./spec/helpers/download_presence_spec.rb[1:3]
./spec/helpers/download_presence_spec.rb[1:1]
./spec/helpers/rake_runner_spec.rb[1:2]
./spec/helpers/bundler_wrapper_spec.rb[1:2]
./spec/hatchet/getting_started_spec.rb[1:1]
./spec/helpers/stale_file_cleaner_spec.rb[1:1]
./spec/helpers/ruby_version_spec.rb[1:1]
./spec/helpers/outdated_ruby_version_spec.rb[1:1]
./spec/rake/deploy_check_spec.rb[1:1:2]
./spec/helpers/shell_spec.rb[1:1:1]
./spec/unit/bash_functions_spec.rb[1:1]
./spec/rake/deploy_check_spec.rb[1:2:7]
./spec/helpers/outdated_ruby_version_spec.rb[1:7]
./spec/hatchet/ruby_spec.rb[1:8:1:1]
./spec/helpers/outdated_ruby_version_spec.rb[1:2]
./spec/helpers/fetcher_spec.rb[1:1]
./spec/helpers/bundler_wrapper_spec.rb[1:1]
./spec/helpers/config_spec.rb[1:1]
./spec/helpers/shell_spec.rb[1:3:1]
./spec/helpers/shell_spec.rb[1:4:1]
./spec/hatchet/ruby_spec.rb[1:1:1]
./spec/hatchet/rails7_spec.rb[1:1]
./spec/helpers/rails_runner_spec.rb[1:4]
./spec/hatchet/node_spec.rb[1:2]
./spec/unit/bash_functions_spec.rb[1:2]
./spec/hatchet/rails4_spec.rb[1:1]
./spec/rake/deploy_check_spec.rb[1:2:8]
./spec/helpers/bundler_wrapper_spec.rb[3:2]
./spec/helpers/outdated_ruby_version_spec.rb[1:4]
./spec/helpers/shell_spec.rb[1:5:1:1]
./spec/helpers/binstub_check_spec.rb[1:3]
./spec/helpers/rails_runner_spec.rb[1:3]
./spec/hatchet/rails4_spec.rb[1:2]
./spec/helpers/ruby_version_spec.rb[1:6]
./spec/hatchet/rails6_spec.rb[1:2]
./spec/hatchet/rubies_spec.rb[3:1]
./spec/hatchet/ci_spec.rb[1:3]
./spec/hatchet/rubies_spec.rb[1:1]
./spec/hatchet/bundler_spec.rb[1:2]
./spec/helpers/download_presence_spec.rb[1:5]
./spec/helpers/stale_file_cleaner_spec.rb[1:2]
./spec/helpers/download_presence_spec.rb[1:6]
./spec/hatchet/ruby_spec.rb[4:1]
./spec/helpers/yarn_installer_spec.rb[1:1:1]
./spec/helpers/bundler_wrapper_spec.rb[4:1]
./spec/helpers/bundler_wrapper_spec.rb[2:1]
./spec/installers/heroku_ruby_installer_spec.rb[1:2:1]
./spec/helpers/ruby_version_spec.rb[1:3]
./spec/rake/deploy_check_spec.rb[1:2:4]
./spec/hatchet/getting_started_spec.rb[1:2]
./spec/rake/deploy_check_spec.rb[1:1:1]
./spec/rake/deploy_check_spec.rb[1:2:2]
./spec/rake/deploy_check_spec.rb[1:1:3]
./spec/helpers/ruby_version_spec.rb[1:5]
./spec/helpers/binstub_check_spec.rb[1:4]
./spec/hatchet/rails7_spec.rb[1:3]
./spec/hatchet/ruby_spec.rb[1:4:1]
./spec/helpers/rails_runner_spec.rb[1:5]
./spec/helpers/download_presence_spec.rb[1:4]
./spec/helpers/binstub_check_spec.rb[1:5]
./spec/helpers/node_installer_spec.rb[1:1:1]
./spec/hatchet/ci_spec.rb[1:1]
./spec/hatchet/ci_spec.rb[1:5]
./spec/hatchet/ruby_spec.rb[1:2:3]
./spec/helpers/bundler_wrapper_spec.rb[4:3:1]
./spec/hatchet/bugs_spec.rb[1:1:1]
./spec/rake/deploy_check_spec.rb[1:2:6]
./spec/rake/deploy_check_spec.rb[1:2:3]
./spec/helpers/shell_spec.rb[1:3:2]
./spec/helpers/shell_spec.rb[1:5:1:2]
./spec/helpers/rake_runner_spec.rb[1:5]
./spec/hatchet/buildpack_spec.rb[1:1]
./spec/hatchet/ruby_spec.rb[3:1]
./spec/hatchet/ruby_spec.rb[1:2:1]
./spec/helpers/binstub_check_spec.rb[1:1]
./spec/hatchet/ruby_spec.rb[1:6:1]
./spec/helpers/bundler_wrapper_spec.rb[4:2]
./spec/helpers/outdated_ruby_version_spec.rb[1:6]
./spec/helpers/ruby_version_spec.rb[1:2]
./spec/hatchet/ruby_spec.rb[1:2:2]
./spec/hatchet/ruby_spec.rb[1:7:1:1]
./spec/rake/deploy_check_spec.rb[1:2:1]
./spec/hatchet/ruby_spec.rb[1:3:1]
./spec/installers/heroku_ruby_installer_spec.rb[1:1:1]
./spec/helpers/download_presence_spec.rb[1:7]
./spec/helpers/download_presence_spec.rb[1:9]
./spec/hatchet/ruby_spec.rb[1:5:1]
./spec/hatchet/ci_spec.rb[1:2]
./spec/helpers/shell_spec.rb[1:2:1]
./spec/helpers/download_presence_spec.rb[1:8]
./spec/helpers/outdated_ruby_version_spec.rb[1:5]
./spec/hatchet/ruby_spec.rb[5:1]
./spec/hatchet/ruby_spec.rb[1:8:2:1]
./spec/helpers/bundler_wrapper_spec.rb[3:1]
./spec/helpers/rails_runner_spec.rb[1:2]
./spec/hatchet/ruby_spec.rb[2:1]
./spec/hatchet/node_spec.rb[1:1]
./spec/helpers/node_installer_spec.rb[1:1:2]
./spec/helpers/rails_runner_spec.rb[1:6]
./spec/rake/deploy_check_spec.rb[1:2:9]
./spec/helpers/rake_runner_spec.rb[1:4]
./spec/hatchet/rails3_spec.rb[1:1]
⛄️ 3.1.6 🚀 /Users/rschneeman/Documents/projects/work/heroku-buildpack-ruby (jwl/update-nodejs-default)
$
schneems commented 1 week ago

Weirdly it seems that it's executing the node installer spec. Manually executing it produces a failure:

$ be rspec "./spec/helpers/node_installer_spec.rb[1:1:2]"
Run options: include {:focused=>true, :ids=>{"./spec/helpers/node_installer_spec.rb"=>["1:1:2"]}}
       Command: 'set -o pipefail; curl -L --fail --retry 5 --retry-delay 1 --connect-timeout 3 --max-time 30 https://nodejs.org/download/release/v22.110/node-v22.110-linux-arm64.tar.gz -s -o - | tar zxf - node-v22.110-linux-arm64/bin/node' failed on attempt 1 of 3.
       Command: 'set -o pipefail; curl -L --fail --retry 5 --retry-delay 1 --connect-timeout 3 --max-time 30 https://nodejs.org/download/release/v22.110/node-v22.110-linux-arm64.tar.gz -s -o - | tar zxf - node-v22.110-linux-arm64/bin/node' failed on attempt 2 of 3.
F

Failures:

  1) LanguagePack::Helpers::NodeInstaller#install should extract a node binary on arm64
     Failure/Error: raise error_class, "Command: '#{command}' failed unexpectedly:\n#{result}"

     LanguagePack::Fetcher::FetchError:
       Command: 'set -o pipefail; curl -L --fail --retry 5 --retry-delay 1 --connect-timeout 3 --max-time 30 https://nodejs.org/download/release/v22.110/node-v22.110-linux-arm64.tar.gz -s -o - | tar zxf - node-v22.110-linux-arm64/bin/node' failed unexpectedly:
       tar: node-v22.110-linux-arm64/bin/node: Not found in archive
       tar: Error exit delayed from previous errors.
     # ./lib/language_pack/shell_helpers.rb:97:in `block in run!'
     # ./lib/language_pack/shell_helpers.rb:91:in `times'
     # ./lib/language_pack/shell_helpers.rb:91:in `run!'
     # ./lib/language_pack/fetcher.rb:31:in `fetch_untar'
     # ./lib/language_pack/helpers/node_installer.rb:24:in `block (2 levels) in install'
     # ./lib/language_pack/helpers/node_installer.rb:23:in `chdir'
     # ./lib/language_pack/helpers/node_installer.rb:23:in `block in install'
schneems commented 1 day ago

The codepath affected in #1516 is exercised when running 

$ be rspec spec/helpers
# ...
  1) LanguagePack::Helpers::YarnInstaller#install should extract the yarn package
     Failure/Error: FileUtils.mv(File.join(dir, name), name)

     Errno::ENOENT:
       No such file or directory @ rb_file_s_rename - (/var/folders/yr/yytf3z3n3q336f1tj2b2j0gw0000gn/T/d20241111-85302-vlehpy/yarn-v1.22.22, yarn-v1.22.22)
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:541:in `rename'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:541:in `block in mv'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:1577:in `block in fu_each_src_dest'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:1593:in `fu_each_src_dest0'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:1575:in `fu_each_src_dest'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/fileutils.rb:532:in `mv'
     # ./lib/language_pack/helpers/yarn_installer.rb:27:in `block in install'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/tmpdir.rb:96:in `mktmpdir'
     # ./lib/language_pack/helpers/yarn_installer.rb:22:in `install'
     # ./spec/helpers/yarn_installer_spec.rb:10:in `block (5 levels) in <top (required)>'
     # ./spec/helpers/yarn_installer_spec.rb:8:in `chdir'
     # ./spec/helpers/yarn_installer_spec.rb:8:in `block (4 levels) in <top (required)>'
     # /Users/rschneeman/.rubies/ruby-3.1.6/lib/ruby/3.1.0/tmpdir.rb:96:in `mktmpdir'
     # ./spec/helpers/yarn_installer_spec.rb:7:in `block (3 levels) in <top (required)>'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `instance_exec'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `block in run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `block in run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:124:in `block in run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `loop'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:110:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-retry-0.6.2/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-retry-0.6.2/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:259:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `map'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `run_examples'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:607:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `map'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'
     # /Users/rschneeman/.gem/ruby/3.1.6/bin/rspec:25:in `load'
     # /Users/rschneeman/.gem/ruby/3.1.6/bin/rspec:25:in `<top (required)>'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
     # /Users/rschneeman/.gem/ruby/3.1.6/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'
     # /Users/rschneeman/.gem/ruby/3.1.6/bin/bundle:25:in `load'
     # /Users/rschneeman/.gem/ruby/3.1.6/bin/bundle:25:in `<main>'

Finished in 36.97 seconds (files took 0.39909 seconds to load)
61 examples, 1 failure

Failed examples:

rspec ./spec/helpers/yarn_installer_spec.rb:6 # LanguagePack::Helpers::YarnInstaller#install should extract the yarn package

Had this issue with ci-queue been fixed, it would have prevented that change from going into prod.

schneems commented 1 day ago

Seems like the problem is that CI isn't running ANY tests:

-----> Running test command `bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue $REDIS_URL --format documentation || { cat log/test_order.log;  $(exit 1); }`...
Finished in 9.72 seconds (files took 20.02 seconds to load)
0 examples, 0 failures
-----> test command `bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue $REDIS_URL --format documentation || { cat log/test_order.log;  $(exit 1); }` completed successfully
schneems commented 1 day ago

It seems to be related to https://devcenter.heroku.com/changelog-items/2992. In which redis connections are now SSL by default, however they need to be set to OpenSSL::SSL::VERIFY_NONE (https://devcenter.heroku.com/articles/connecting-heroku-redis) as the certificate is self signed.

When I force it to use a non-ssl version of Redis it works:

$ heroku ci:debug -p <redacted>
# ...
~ $ export REDIS_URL=$REDIS_TEMPORARY_URL
$ bundle exec rspec-queue --max-requeues=3 --timeout 180 --queue $REDIS_URL --format documentation || { cat log/test_order.log;  $(exit 1); }
Worker electected as leader, pushing 118 tests to the queue.

Finished pushing 118 tests to the queue in 0.01s.
-----> Installing bundler 2.3.25
-----> Removing BUNDLED WITH version in the Gemfile.lock
should detect successfully
converts `bundle platform --ruby` for released versions
Hatchet setup: "hatchet-t-20528cbb54" for "default_ruby"
Destroying "hatchet-t-20528cbb54": ff4d7def-a7ad-4926-a391-0c22fdd0b76a, (0/100) reason: teardown
should not overwrite already set environment variables
-----> Installing bundler 1.17.3
-----> Removing BUNDLED WITH version in the Gemfile.lock
handles apps with ruby versions locked in Gemfile.lock
Hatchet setup: "hatchet-t-4ad6040dba" for "default_ruby"
Destroying "hatchet-t-4ad6040dba": 80b9e96e-9538-4532-91e7-6da40b972175, (0/100) reason: teardown
loads custom rake binstub
Hatchet setup: "hatchet-t-9baa19c1b8" for "sqlite3_gemfile"
^C
RSpec is shutting down and will print the summary report... Interrupt again to force quit (warning: at_exit hooks will be skipped if you force quit).
Destroying "hatchet-t-9baa19c1b8": 0e746c4b-2f96-4930-95a6-8fc1d0c89492, (0/100) reason: teardown
should raise on sqlite3

Finished in 55.04 seconds (files took 0.42277 seconds to load)
6 examples, 0 failures