search

oli-obk / ui_test

A test framework for testing rustc diagnostics output
24 stars 24 forks source link

Feature suggestion: Comment annotation for execution behavior #105

Open cedihegi opened 1 year ago

cedihegi commented 1 year ago

Hey, so for my use-case it would be really useful if these comment annotations were also possible to be used to specify the runtime behavior of the generated executables. In particular, comments of the form //~PANIC: XXX to specify that a file panics on a specific line, containing XXX in its error message. I am willing to implement this, but I just wanted to ask first what you think about this and whether this would be accepted. Cheers :)

oli-obk commented 1 year ago

The main issue with this is parsing the output of the program to look for panics. With rustc we get nice json output... Tho I realize now that's only until rustc ICEs. And ICE tests are already something that works badly in the rustc test suite.

I guess we could add a parser specifically for the default Rust panic format (tho we'll probably have to add a way to override it, to allow non-default panic handlers and arbitrary non-panic failure parsers).

I'll think about how to best do this (suggestions are welcome), as I've wanted to allow people to use something other than rustc and not have to match the json output, but instead be able to provide their own parser.

I don't want to spread Box<dyn Fn(SomeInput) -> SomeOutput> in arguments and fields everywhere, so I need to figure out how to split this crate up further into lego-like components. As an example of what I mean, look at https://github.com/oli-obk/ui_test/pull/92 which made the threading logic fully independent of the actual test suite.

oli-obk commented 5 months ago

This should be implementable now. //@run is just a custom post-test-compiled-successfully handler now