Improve stability of the test suite

[ayushn21]

Recently, I've found I need to re-run our test suite several times in PRs to get it to go green. I'm not sure what's caused the recent jump in false failures but I think it's something we should sort out sooner rather than later.

I get a fair amount of false failures locally as well.

Unfortunately I'm not sure what's causing the false negatives or how to investigate and improve it. I thought I'd open an issue for it so we can track it.

[jaredcwhite]

I've seen that as well. It seems like something related to an autoloader issue (aka it's not doing the proper cascade of local content overriding plugin content). I've never been able to reproduce it locally though, only in GH actions. :(

[ayushn21]

It's not just that issue, there's quite a few random "crashes" happening as well ... there aren't any failing tests logged out, just says the command failed

+ ruby -S bundle exec rake TESTOPTS=--profile test
23
/opt/hostedtoolcache/Ruby/3.0.0/x64/bin/ruby -w -I"lib:lib:test" /home/runner/work/bridgetown/bridgetown/vendor/bundle/ruby/3.0.0/gems/rake-13.0.3/lib/rake/rake_test_loader.rb "test/fixtures/test_automation.rb" "test/test_ansi.rb" "test/test_apply_command.rb" "test/test_cleaner.rb" "test/test_collections.rb" "test/test_commands_serve_servlet.rb" "test/test_components.rb" "test/test_configuration.rb" "test/test_configure_command.rb" "test/test_data_reader.rb" "test/test_defaults_reader.rb" "test/test_doctor_command.rb" "test/test_document.rb" "test/test_drop.rb" "test/test_entry_filter.rb" "test/test_erb.rb" "test/test_excerpt.rb" "test/test_excerpt_drop.rb" "test/test_filters.rb" "test/test_front_matter_defaults.rb" "test/test_generated_page.rb" "test/test_generated_site.rb" "test/test_kramdown.rb" "test/test_layout_reader.rb" "test/test_liquid_extensions.rb" "test/test_liquid_renderer.rb" "test/test_log_adapter.rb" "test/test_new_command.rb" "test/test_page.rb" "test/test_path_sanitization.rb" "test/test_plugin_manager.rb" "test/test_post_reader.rb" "test/test_regenerator.rb" "test/test_related_posts.rb" "test/test_relations.rb" "test/test_resource.rb" "test/test_ruby_helpers.rb" "test/test_serve_command.rb" "test/test_site.rb" "test/test_site_drop.rb" "test/test_static_file.rb" "test/test_tags.rb" "test/test_url.rb" "test/test_utils.rb" "test/test_validatable.rb" "test/test_webpack_command.rb" "test/test_yaml_parser.rb" --profile
24

25
# Running tests with run options --profile --seed 16465:
26

27
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Coverage report generated for Unit Tests to /home/runner/work/bridgetown/bridgetown/bridgetown-core/coverage. 7751 / 9933 LOC (78.03%) covered.
28
rake aborted!
29
Command failed with status (1): [ruby -w -I"lib:lib:test" /home/runner/work/bridgetown/bridgetown/vendor/bundle/ruby/3.0.0/gems/rake-13.0.3/lib/rake/rake_test_loader.rb "test/fixtures/test_automation.rb" "test/test_ansi.rb" "test/test_apply_command.rb" "test/test_cleaner.rb" "test/test_collections.rb" "test/test_commands_serve_servlet.rb" "test/test_components.rb" "test/test_configuration.rb" "test/test_configure_command.rb" "test/test_data_reader.rb" "test/test_defaults_reader.rb" "test/test_doctor_command.rb" "test/test_document.rb" "test/test_drop.rb" "test/test_entry_filter.rb" "test/test_erb.rb" "test/test_excerpt.rb" "test/test_excerpt_drop.rb" "test/test_filters.rb" "test/test_front_matter_defaults.rb" "test/test_generated_page.rb" "test/test_generated_site.rb" "test/test_kramdown.rb" "test/test_layout_reader.rb" "test/test_liquid_extensions.rb" "test/test_liquid_renderer.rb" "test/test_log_adapter.rb" "test/test_new_command.rb" "test/test_page.rb" "test/test_path_sanitization.rb" "test/test_plugin_manager.rb" "test/test_post_reader.rb" "test/test_regenerator.rb" "test/test_related_posts.rb" "test/test_relations.rb" "test/test_resource.rb" "test/test_ruby_helpers.rb" "test/test_serve_command.rb" "test/test_site.rb" "test/test_site_drop.rb" "test/test_static_file.rb" "test/test_tags.rb" "test/test_url.rb" "test/test_utils.rb" "test/test_validatable.rb" "test/test_webpack_command.rb" "test/test_yaml_parser.rb" --profile]
30
/home/runner/work/bridgetown/bridgetown/vendor/bundle/ruby/3.0.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
31
/opt/hostedtoolcache/Ruby/3.0.0/x64/bin/bundle:23:in `load'
32
/opt/hostedtoolcache/Ruby/3.0.0/x64/bin/bundle:23:in `<main>'
33
Tasks: TOP => test
34
(See full trace by running task with --trace)
35

36
real    0m58.149s
37
user    0m42.798s
38
sys 0m3.848s
39
rake aborted!
40
Command failed with status (1): [cd bridgetown-core && script/cibuild...]
41
/home/runner/work/bridgetown/bridgetown/Rakefile:7:in `block in <top (required)>'
42
/home/runner/work/bridgetown/bridgetown/vendor/bundle/ruby/3.0.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
43
/opt/hostedtoolcache/Ruby/3.0.0/x64/bin/bundle:23:in `load'
44
/opt/hostedtoolcache/Ruby/3.0.0/x64/bin/bundle:23:in `<main>'

No sure what's caused this to start happening. My latest PR changes code not covered by unit tests and I've run the suite about 4 times and it's still not green :(

[jaredcwhite]

Yeah it's grown increasingly unstable, and I'm not sure why other than perhaps something related to Zeitwerk autoloading. Should make this a top priority after 0.21 release.

[jaredcwhite]

This random crash thing is happening all the time now…didn't use to do this. Will be digging in shortly.

[jaredcwhite]

So a couple of things:

• By switching to Minitest::Reporters::SpecReporter instead of the default one, we can get a line-by-line play of all the tests run, and by analyzing where it crashes it looks like it's usually one of the command tests (new, webpack, etc.)
• I added an ENV check so those command tests aren't running on GH Actions now in #331, and so far that seems to be a big improvement.

So, it looks like we need to put in effort to see why those action tests are often crashing…but in the meantime I'm tempted to bypass them in GH Actions entirely and only run them locally for the time being.

[jaredcwhite]

@ayushn21 For now I've merged in the change so some of those Thor action tests aren't getting tested in CI. Not ideal but for now much more stable and we should get green checks the majority of the time again.

It's still flaky locally of course, so this is a temporary fix.

[ayushn21]

Cool sounds good. I wonder why it's just those tests crashing. Must be related to a change I made for the Webpack command as it's a fairly recent occurrence. I'll try to dig into it a bit some time this week.