Miscenallenous enhancements

[jmarrec]

• [ ] When running a ton of tests, especially in debug, it'd be nice if the entire suite wouldn't just stop if you hit one OS_ASSERT for eg (probably means putting each indiv run in a try/catch statement)

• [ ] I'm not even sure it's feasible, but it would be awesome if when you hit a C++ crash it could report the ruby line that lead to this.

• [ ] Or least the single file that made it crash, which is possible for sure, and crash aside I'd want this anyway because it'd also help to see the progress of the test suite and understand the outputs you see: right now I see a lot of measure output (**MEASURE APPLICABILITY** etc) in the terminal but I have no clue which one that's for.

• [ ] The ability to rerun only subtests that failed in a previous run would also be very beneficial (eg: if I run the entire nrel_published and only 5 of them failed, I'd want to rerun these 5 easily until they pass without really having to type 5 commands)

• [ ] Ability to sort the dashboard table by column header would be awesome too. Eg: Clik on Test > Fail and see the biggest offenders at the top

[nllong]

1) This is very difficult with how the tests work. That is just the nature of unit testing frameworks.

2) If it is a segfault, then I don't think this is possible. Otherwise, I'm sure we could catch it.

3) Loop into item 2.

4) You will need to look into MiniTest to see if this is possible. I know you can do this with RSpec.

5) Yes! we need this.

[jmarrec]

I guess there's something fundamental I don't know which is how this gem works :)

I know each unit test for measures eventually uses minitest. I assumed, likely wrongly, that when I run openstudio measure -r . That it would basically traverse the directory tree until it found tests files to run via minitest, which would make parallelization and printing the measure name before launching the test easy, as well as putting it in try catch statement in case you hit an assert. Reruning only failed tests would be also feasible since you'd just have to parse any existing test_results folder to find those that passed and skip them.

That's not how it works does it?