Test output missing JSON

[bmulholland]

My test output has two things: the result of the test, and a stdout message about code coverage.

For some reason, only the code coverage stdout message is making it into the output file I seen in the neotest log (and so no results show up in vim). The test results are not in that file at all.

I tried running the command that is being constructed: bundle exec rspec file_spec.rb -f json -o test.json -- and when I do that, the test.json is filled only with the opposite: json and no stdout.

Is the command running via something else that's maybe directing stdout to the output file too, and then stdout is overwriting the rspec output? Can you point me to what the next step after the command construction is? https://github.com/olimorris/neotest-rspec/blob/main/lua/neotest-rspec/init.lua#L115

[olimorris]

I tried running the command that is being constructed: bundle exec rspec file_spec.rb -f json -o test.json -- and when I do that, the test.json is filled only with the opposite: json and no stdout.

For a testing a file, the command is bundle exec rspec -f json -o output.json spec/render_spec.rb. What about if you turn off code coverage? Also, what about if you put spec/file_spec.rb in the command?

I actually have a complex test working with simplecov and all results are being generated okay.

[bmulholland]

Specifically, I'm looking at the file neotest outputs in its log as DEBUG | 2022-06-13T15:05:20Z+0200 | ...ker/start/neotest/lua/neotest/client/strategies/init.lua:37 | Output of process

For a testing a file, the command is bundle exec rspec -f json -o output.json spec/render_spec.rb

I tried this exact command, replacing output.json with the /var/folders/ tmp file from one of the neotest runs (plus a correct spec file) and the output is correctly output into the file.

What about if you turn off code coverage?

Then the output file is blank, so I guess it's not being overwritten?

Also, what about if you put spec/file_spec.rb in the command?

I don't understand this one?

[olimorris]

Can you share a snapshot of the log?

In mine I see:

and I get the following in the /var/folders folder:

# /var/folders/v2/78qsgrqj3c19g1z0vlsxjb3c0000gn/T/nvim475sUp/1
{
  "version": "3.11.0",
  "examples": [
    {
      "id": "./spec/simple_example_spec.rb[1:1:1]",
      "description": "returns 4",
      "full_description": "Some maths calculations passing test returns 4",
      "status": "passed",
      "file_path": "./spec/simple_example_spec.rb",
      "line_number": 3,
      "run_time": 0.000253,
      "pending_message": null
    },
    {
      "id": "./spec/simple_example_spec.rb[1:2:1]",
      "description": "doesn't return 4",
      "full_description": "Some maths calculations failing test doesn't return 4",
      "status": "failed",
      "file_path": "./spec/simple_example_spec.rb",
      "line_number": 8,
      "run_time": 0.006417,
      "pending_message": null,
      "exception": {
        "class": "RSpec::Expectations::ExpectationNotMetError",
        "message": "\nexpected: 5\n     got: 4\n\n(compared using ==)\n",
        "backtrace": [
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-support-3.11.0/lib/rspec/support.rb:102:in `block in <module:Support>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-support-3.11.0/lib/rspec/support.rb:111:in `notify_failure'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/fail_with.rb:35:in `fail_with'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:38:in `handle_failure'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:56:in `block in handle_matcher'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/expectation_target.rb:65:in `to'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-expectations-3.11.0/lib/rspec/expectations/expectation_target.rb:101:in `to'",
          "/Users/Oli/Code/Projects/neotest-rspec/spec/simple_example_spec.rb:9:in `block (3 levels) in <top (required)>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:263:in `instance_exec'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:263:in `block in run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:486:in `block in run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:624:in `run_around_example_hooks_for'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/hooks.rb:486:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example.rb:259:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:642:in `map'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:642:in `run_examples'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:607:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:608:in `block in run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:608:in `map'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/example_group.rb:608:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `map'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/configuration.rb:2068:in `with_suite_hooks'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:116:in `block in run_specs'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/reporter.rb:74:in `report'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:115:in `run_specs'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:89:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:71:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/lib/rspec/core/runner.rb:45:in `invoke'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/rspec-core-3.11.0/exe/rspec:4:in `<top (required)>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/bin/rspec:25:in `load'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/bin/rspec:25:in `<top (required)>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `load'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:58:in `kernel_load'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli/exec.rb:23:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:483:in `exec'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:31:in `dispatch'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/cli.rb:25:in `start'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/bundler-2.3.14/exe/bundle:48:in `block in <top (required)>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/site_ruby/3.1.0/bundler/friendly_errors.rb:103:in `with_friendly_errors'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/lib/ruby/gems/3.1.0/gems/bundler-2.3.14/exe/bundle:36:in `<top (required)>'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/bin/bundle:25:in `load'",
          "/Users/Oli/.asdf/installs/ruby/3.1.1/bin/bundle:25:in `<main>'"
        ]
      }
    }
  ],
  "summary": {
    "duration": 0.007294,
    "example_count": 2,
    "failure_count": 1,
    "pending_count": 0,
    "errors_outside_of_examples_count": 0
  },
  "summary_line": "2 examples, 1 failure"
}

[bmulholland]

Give me a bit, I'm reproducing some of the oddness with rspec manually. I think the issue may have to do with writing to the temp file (specifically, it's writing to the local dir but not to /var/folders)? Will update.

[olimorris]

Sure. Maybe try running the simple example I included in the repo and see if that works for you as a first step.

[bmulholland]

I can repro the problem with the simple_example_spec

I've seen inconsistent behaviour. Sometimes I can repro the issue (output file is blank) when running the command manually. Other times running it manually works. But it never works when run via neotest. I've verified this by taking the actual command output and running that directly.

I have to get to real-work now, so I'll dig in another time.

[alxekb]

Here is my 5 cents 🙂

I really like the idea, many many thanks @olimorris for your effort! Great job! 🥇

I got this tree with the output. Hope it helps 👍🏻

{
  "version":"3.10.1",
  "examples":[
    {
      "id":"./spec/models/customer_spec.rb[1:1:1]",
      "description":"is valid with valid attributes",
      "full_description":"Customer when validating is valid with valid attributes",
      "status":"passed",
      "file_path":"./spec/models/customer_spec.rb",
      "line_number":19,
      "run_time":0.063148,
      "pending_message":null
    },
    {
      "id":"./spec/models/customer_spec.rb[1:1:2]",
      "description":"is invalid without a nickname",
      "full_description":"Customer when validating is invalid without a nickname",
      "status":"passed",
      "file_path":"./spec/models/customer_spec.rb",
      "line_number":23,
      "run_time":0.363921,
      "pending_message":null
    }
  ],
  "summary":{
    "duration":0.746824,
    "example_count":2,
    "failure_count":0,
    "pending_count":0,
    "errors_outside_of_examples_count":0
  },
  "summary_line":"2 examples, 0 failures"
}

[olimorris]

@alxekb thanks for the output! Could you share your tests? I'm just interested in the describe, context and it content`. Also could you share the treesitter ids from the logs? Suspect that is where the error is coming from

[alxekb]

@alxekb thanks for the output! Could you share your tests? I'm just interested in the describe, context and it content`. Also could you share the treesitter ids from the logs? Suspect that is where the error is coming from

PR rised #3

[olimorris]

Hey @bmulholland, just checking in to see if you've had any luck?

[bmulholland]

Well things seem different today, at least. The output file now seems to be (usually?) filled with the rspec result JSON -- but the status of the test doesn't make it to display in neovim:

log and results path file: https://gist.github.com/bmulholland/de6b2eb3447bce0a056661ec5fbd84c0

I'm happy to debug more on my end, I just don't know where to start. Feel free to give me pointers on things to check.

[olimorris]

Unfortunate timing...an update to neotest actually broke the adapter. I've just fixed it now so hoping it may work for you.

If you run lua require("neotest").run.run() next to it (so run the nearest test) and check the logs you should see the adapter output the RSpec id and Treesitter id. When they do not match, that's when we get the weird issues.

[bmulholland]

It works!

[bmulholland]

Thanks for your help. And thanks for the adapter! I haven't the faintest clue what was happening yesterday...

[olimorris]

Awesome!! So glad to hear it. I think you may have been one commit ahead of myself. Enjoy neotest with RSpec 😄 .