search

firstdraft / appdev_template

A Rails template for generating homework projects
0 stars 1 forks source link

Show ruby error message in test summary line #173

Closed jelaniwoods closed 3 years ago

jelaniwoods commented 3 years ago

Resolves https://github.com/firstdraft/grade_runner/issues/44

To run locally, you can clone any appdev project and replace the json_output_formatter.rb with these updates and then run

bundle exec rspec --require ./spec/support/json_output_formatter.rb --format JsonOutputFormatter --no-color

You could alternatively run rails grade with any active token and view the results in the grades UI.

raghubetina commented 3 years ago

@jelaniwoods Do you think some sort of truncating would be helpful? For me the output is quite long.

raghubetina commented 3 years ago

Looking through, I see ":exception":{":class":"NameError",":message":"uninitialized constant Director". I wonder if that would be consistently there in these cases, if we could parse the JSON and only display that part, and whether that would be an improvement for the students.

jelaniwoods commented 3 years ago

@raghubetina

Looking through, I see ":exception":{":class":"NameError",":message":"uninitialized constant Director". I wonder if that would be consistently there in these cases, if we could parse the JSON and only display that part, and whether that would be an improvement for the students.

@raghubetina Ah, I see. In my testing, this :exception hash doesn't appear to be populated if an error occurs outside of the examples.

https://github.com/rspec/rspec-core/blob/ecb65d2d4ea7b381472e3084f45e2da94e00ce91/lib/rspec/core/formatters/json_formatter.rb#L35-L48

We can customize the summary line for regular errors as well if that would be helpful?

In the "error occurred outside of examples" case, we only have the messages Array, which isn't formatted very well.

messages
[
    [0] "\nAn error occurred while loading ./spec/models/models_spec.rb.\nFailure/Error: Director\n\nNameError:\n  uninitialized constant Director\n# ./spec/models/models_spec.rb:2:in `<main>'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n"
]
{":version":"3.9.3",":messages":["\nAn error occurred while loading ./spec/models/models_spec.rb.\nFailure/Error: Director\n\nNameError:\n  uninitialized constant Director\n# ./spec/models/models_spec.rb:2:in `<main>'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n"],":examples":[],":summary":{":duration":7.1e-05,":example_count":0,":errors_outside_of_examples_count":1,":failure_count":0,":pending_count":0,":total_points":0,":earned_points":0,":score":0},":summary_line":"0 tests, 0 failures, 0/0 points, An error occurred while running tests\nAn error occurred while loading ./spec/models/models_spec.rb.\nFailure/Error: Director\n\nNameError:\n  uninitialized constant Director\n# ./spec/models/models_spec.rb:2:in `<main>'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n# /Users/jelani/.rvm/gems/ruby-2.6.6/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:59:in `load'\n"}%

It doesn't seem as standard so I wasn't sure how to truncate it.

raghubetina commented 3 years ago

@jelaniwoods Ah I gotcha. Perhaps we can make a card to consider customizing the summary line for regular errors; for this outside of examples case, LGTM :ship: