Running rails tests in the vscode test explorer is quite slow

andypeters commented 3 months ago

Description:

When running a test class in the VScode Test Explorer, the tests take quite a while to complete. It seems they are running one at a time starting with the last test first (visually from the bottom up). Additionally, it indicates running in a single process per test.

If this is known or expected, I'm sorry I missed the mention. I scoped out the issue list here and shopify/ruby-lsp-rails as best I could. There were a few somewhat related regarding speed to rspec tests, but I know that is an extension so I thought it fitting to report in regard to minitest here. One being this: https://github.com/Shopify/ruby-lsp/issues/2049

Reproduction steps

Open any rails project in VScode.
Select a test. ie: test/models/foo_test.
Wait for Run / Run in Terminal / Debug appear.
Open Test Explorer from the sidebar. Wait for the class to be visible (ie step three of course).
Either click the "Play icon" next to the test class or click "Run" in the editor. (I have gifs illustrating below)
Observe the tests run one at a time. Additionally, watch "Run in Terminal" zip by.

test_helper.rb:

parallelize(workers: :number_of_processors)

Grasping at straws, I went bonkers and nuclear 💥

Removed all of .vscode: rm -rf ~/.vscode
Deleted VSCode. Reinstalled it.
Deleted everything in Library/Application Support/Code. (or is it visual studio code - can't remember because its gone 😆 )
When reinstalling VScode, I did not sync my settings and only installed the ruby-lsp plugin.

Gifs for ya from a throw away rails app

Test Explorer	Console

Screenshot for "running in a single process"

Using very basic test. ie:

 test "test 1" do
    assert false
  end

My environment

FWIW, this is the result on a handful of projects which are all up-to-date with Rails 7 using minitest. I'm using asdf and for this throw away project ruby 3.3.2.

Here are the details for the throwaway rails project:

Output from VSCode for ruby-lsp:

2024-08-01 10:12:17.800 [info] (vscode-ruby-lsp-tester) Discovered version manager asdf
2024-08-01 10:12:17.801 [info] (vscode-ruby-lsp-tester) Running command: `. /opt/homebrew/opt/asdf/libexec/asdf.sh && asdf exec ruby -W0 -rjson -e 'STDERR.print({env: ENV.to_h,yjit:!!defined?(RubyVM::YJIT),version:RUBY_VERSION}.to_json)'` in /Users/andy/Developer/vscode-ruby-lsp-tester using shell: /bin/zsh
2024-08-01 10:12:18.150 [info] (vscode-ruby-lsp-tester) Ruby LSP> Running bundle install for the custom bundle. This may take a while...
Ruby LSP> Command: (bundle check || bundle install) 1>&2

2024-08-01 10:12:18.270 [info] (vscode-ruby-lsp-tester) Bundler can't satisfy your Gemfile's dependencies.
Install missing gems with `bundle install`.

2024-08-01 10:12:18.764 [info] (vscode-ruby-lsp-tester) Fetching gem metadata from https://rubygems.org/
2024-08-01 10:12:18.764 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.826 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.847 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.864 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.889 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.896 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.899 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.900 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:18.900 [info] (vscode-ruby-lsp-tester) .
2024-08-01 10:12:19.299 [info] (vscode-ruby-lsp-tester) 
2024-08-01 10:12:19.323 [info] (vscode-ruby-lsp-tester) Resolving dependencies...
2024-08-01 10:12:19.683 [info] (vscode-ruby-lsp-tester) Fetching sorbet-runtime 0.5.11504
2024-08-01 10:12:19.869 [info] (vscode-ruby-lsp-tester) Installing sorbet-runtime 0.5.11504
2024-08-01 10:12:19.882 [info] (vscode-ruby-lsp-tester) Fetching ruby-lsp-rails 0.3.11
2024-08-01 10:12:19.900 [info] (vscode-ruby-lsp-tester) Installing ruby-lsp-rails 0.3.11
2024-08-01 10:12:27.695 [info] (vscode-ruby-lsp-tester) Bundle complete! 17 Gemfile dependencies, 84 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
2024-08-01 10:12:28.158 [info] (vscode-ruby-lsp-tester) Initializing Ruby LSP v0.17.10...
2024-08-01 10:12:28.180 [info] (vscode-ruby-lsp-tester) Finished initializing Ruby LSP!
2024-08-01 10:12:28.209 [info] (vscode-ruby-lsp-tester) Activating Ruby LSP Rails addon v0.3.11
2024-08-01 10:12:28.212 [info] (vscode-ruby-lsp-tester) Ruby LSP Rails booting server
2024-08-01 10:12:32.165 [info] (vscode-ruby-lsp-tester) Finished booting Ruby LSP Rails server

Gem list for lsp:

ruby-lsp (0.17.10)
ruby-lsp-rails (0.3.11)

Bundle list (just in case you want more details and a longer description :-) ):

➜  vscode-ruby-lsp-tester git:(main) ✗ bundle update rails
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Using rack 3.1.7 (was 2.2.9)
Using activesupport 7.1.3.4 (was 7.0.8.4)
Using activemodel 7.1.3.4 (was 7.0.8.4)
Using actionview 7.1.3.4 (was 7.0.8.4)
Using activerecord 7.1.3.4 (was 7.0.8.4)
Using actionpack 7.1.3.4 (was 7.0.8.4)
Using activejob 7.1.3.4 (was 7.0.8.4)
Using actioncable 7.1.3.4 (was 7.0.8.4)
Using activestorage 7.1.3.4 (was 7.0.8.4)
Using actionmailer 7.1.3.4 (was 7.0.8.4)
Using railties 7.1.3.4 (was 7.0.8.4)
Using actionmailbox 7.1.3.4 (was 7.0.8.4)
Using actiontext 7.1.3.4 (was 7.0.8.4)
Using rails 7.1.3.4 (was 7.0.8.4)
Bundle updated!
➜  vscode-ruby-lsp-tester git:(main) ✗ bundle list
Gems included by the bundle:
  * actioncable (7.1.3.4)
  * actionmailbox (7.1.3.4)
  * actionmailer (7.1.3.4)
  * actionpack (7.1.3.4)
  * actiontext (7.1.3.4)
  * actionview (7.1.3.4)
  * activejob (7.1.3.4)
  * activemodel (7.1.3.4)
  * activerecord (7.1.3.4)
  * activestorage (7.1.3.4)
  * activesupport (7.1.3.4)
  * addressable (2.8.7)
  * base64 (0.2.0)
  * bigdecimal (3.1.8)
  * bindex (0.8.1)
  * bootsnap (1.18.3)
  * builder (3.3.0)
  * capybara (3.40.0)
  * concurrent-ruby (1.3.3)
  * connection_pool (2.4.1)
  * crass (1.0.6)
  * date (3.3.4)
  * debug (1.9.2)
  * drb (2.2.1)
  * erubi (1.13.0)
  * globalid (1.2.1)
  * i18n (1.14.5)
  * importmap-rails (2.0.1)
  * io-console (0.7.2)
  * irb (1.14.0)
  * jbuilder (2.12.0)
  * logger (1.6.0)
  * loofah (2.22.0)
  * mail (2.8.1)
  * marcel (1.0.4)
  * matrix (0.4.2)
  * mini_mime (1.1.5)
  * minitest (5.24.1)
  * msgpack (1.7.2)
  * mutex_m (0.2.0)
  * net-imap (0.4.14)
  * net-pop (0.1.2)
  * net-protocol (0.2.2)
  * net-smtp (0.5.0)
  * nio4r (2.7.3)
  * nokogiri (1.16.7)
  * psych (5.1.2)
  * public_suffix (6.0.1)
  * puma (5.6.8)
  * racc (1.8.1)
  * rack (3.1.7)
  * rack-session (2.0.0)
  * rack-test (2.1.0)
  * rackup (2.1.0)
  * rails (7.1.3.4)
  * rails-dom-testing (2.2.0)
  * rails-html-sanitizer (1.6.0)
  * railties (7.1.3.4)
  * rake (13.2.1)
  * rdoc (6.7.0)
  * redis (4.8.1)
  * regexp_parser (2.9.2)
  * reline (0.5.9)
  * rexml (3.3.4)
  * rubyzip (2.3.2)
  * selenium-webdriver (4.23.0)
  * sprockets (4.2.1)
  * sprockets-rails (3.5.2)
  * sqlite3 (1.7.3)
  * stimulus-rails (1.3.3)
  * stringio (3.1.1)
  * strscan (3.1.0)
  * thor (1.3.1)
  * timeout (0.4.1)
  * turbo-rails (2.0.6)
  * tzinfo (2.0.6)
  * web-console (4.2.1)
  * webrick (1.8.1)
  * websocket (1.2.11)
  * websocket-driver (0.7.6)
  * websocket-extensions (0.1.5)
  * xpath (3.2.0)
  * zeitwerk (2.6.17)
Use `bundle info` to print more detailed information about a gem

andyw8 commented 3 months ago

Hi @andypeters, this is a known limitation, there is some work happening here to try improve it:

https://github.com/Shopify/ruby-lsp/pull/2230

(I moved the issue to ruby-lsp-rails, but I'm going to move it back since this is a more general problem)

andypeters commented 3 months ago

Hey thanks @andyw8! Dang, I missed looking at active pull requests. My instinct told me you all knew about it, but you never know. All good and happy to hear you all are trying to get it working. I'll subscribe to that pull request and follow along.

github-actions[bot] commented 1 month ago

This issue is being marked as stale because there was no activity in the last 2 months

malcolmohare commented 1 week ago

I took a look at what it would take to do this, wanted to leave my thoughts here as a guide to either myself picking this up at some point in the future or someone else motivated to give it a try.

Running tests as a single combined process is not hard (at least for rspec, i assume its the same for minitest). It's just providing a different line reference in the run command. The rspec addon/plugin/whatever already provides the command for the test groups, they just get skipped because those TestRun instances don't have the 'example' tag on them. The request for running the tests includes the specific TestRun thats associated with the codelens action, so you can update the code to just execute that command and not traverse the children looking for examples.
The real issue with implementing this is parsing the output (and applying it to the TestRun items to make the UI show the results). And this comes up in multiple ways. It's easy to attach the output to a test when its just a single test per process. But when you run them together, you need to pull apart the output. Since different runners format the output differently, we'd need different parsers. Currently the parsing occurs on the client side of the extension, which doesn't have addon support like the server does (so rspec parsing would have to live apart from its addon repo). And of course, even the test output can be unexpected, if the project uses custom formatters. So no matter how good the parser is that gets shipped, it will have holes.

I think we'd get better mileage by trying to run the tests in parallel. Or investigating whether something like spring could be used to bypass the startup time of the tests.

Shopify / ruby-lsp