Telling tests with duplicate names apart

[auduchinok]

Printers like beforeEach and passed accept test names which specify the tests. When using this technique it's not always possible to tell the tests apart, e.g. when tests don't have a name at all (thus, null is passed to printers) or duplicate names are allowed and used.

From a glance, passing FlatTest instances instead of the names could solve this problem as it'd be possible to inspect the inner TestCode fields and the test names would still be accessible as well. However, it's possible to use the same test code inside different test cases so still leaves a chance of ambiguity. It seems assigning a unique id to each flat test in toTestCodeList could solve it. Another printer accepting a list of flat tests to run could be added as well.

If we're to change the printers and flat tests this way it'd be quite a serious breaking change, though, it seems it could fix all ambiguities in printers.

[haf]

Slightly OT; I'm currently looking at the existing Jaeger support in Logary and how it's being tested. The linked code showcases a pattern I keep coming back to; making single-purpose test functions that delegate to Expecto. I've been using GoLand for a bit, and so I've seen how JetBrains treats testing.T; and it's only possible to run "suites" of t.Run(..., func(...) { THIS IS RUN }).

I was thinking whether it wouldn't be possible to somehow tell the IDE that "here" you can find an entrypoint (like the linked testCaseTarget), either by convention testXXX or otherwise, where there's an attribute defined. Perhaps this is not even visible until one single run of the tests has completed, to have all call-sites of testCase or the like, executed.

I'm asking this; because I don't fully understand how the IDE finds and executes tests. Is your plan to execute the main method, hooking the test printers — how do you find your way back to the test — could you do it if you have the TestCode, because its AST can be looked up? We already have TestRunSummary and that uses FlatTest, so in that sense, the horse has already bolted the stable and you might as well rely on the FlatTest's data in a printer.

[auduchinok]

We already have TestRunSummary and that uses FlatTest, so in that sense, the horse has already bolted the stable and you might as well rely on the FlatTest's data in a printer.

It'd be great to update the UI for tests run results as they appear for individual tests, not after the whole run has finished.

I'm asking this; because I don't fully understand how the IDE finds and executes tests. Is your plan to execute the main method, hooking the test printers — how do you find your way back to the test — could you do it if you have the TestCode, because its AST can be looked up?

I currently look for declarations with Expecto attributes to find the tests. I'm still thinking about approach to get the individual inner tests back (and wanted to try using TestCode, right).

[auduchinok]

I've just thought about a workaround we could use without changing Expecto: I can map the test tree in a way that unique ids I'm interested in are encoded in the test names (and removed in the UI). Or create flat tests manually in a function similar to toTestCodeList as I traverse the test trees anyways. I'll try doing it first before proposing these breaking changes. If you think such changes would be better in the long term, though, I'm happy to discuss them further.

[haf]

@auduchinok I think you'll be having the most insight as you'll have the whole AST-editor-code-runtime mapping in your head as you develop. What I would like to know are things that would allow this test lib able to 'talk to' the IDE; in the end this is a question about a protocol between an AST/runtime and the IDE. The reason I included all those printers was so that live information could be printed as the tests are run.

Anecdotally, I heard from some Scala compiler guys that they had a hard time performing some refactorings/improvements because the IDE was accessing the AST after a number of transformations; in this anecdote's case they'd converted all folds to loops and couldn't go backward to suggest improvements to the source. This anecdote might indicate that keeping a close mapping between what the IDE shows and the original AST. However, there's also some amount of design-time evaluation needed, as discussed above, since a unique test identity (source line location?) and discovery of the tests is needed. Ruby/RSpec does a good job of the source-linenumber-to-runtime mapping, and you can accurately pinpoint any test in the whole test suite.

Design-time evaluation aside; a well written test suite doesn't actually crash or perform any tests while the TestCode's are being evaluated (this is why GoLand can evaluate one level of testing.T callbacks I presume). Having that contract with the IDE's user is probably a good idea; I've had programmers employed who perform a whole lot of setup just to generate the Test values, and in the end that led to a much larger maintenance burden.

So if you can think of good abstractions or protocols, tell me and we can discuss it.

Or create flat tests manually in a function similar to toTestCodeList as I traverse the test trees anyways. I'll try doing it first before proposing these breaking changes.

— tell me how it goes!

[auduchinok]

Even though we can see the whole picture with AST and compiled tests there's still no reliable way to make such a mapping for test elements that are produced at runtime. Imagine creating a test case with a name returned from some arbitrary function: there's no way to find that name at design time. Or even getting a test case via a function call: we'd probably know there're some test tree coming at a particular place but won't be able to analyze the tree by looking at the AST and resolved symbol usages and to find the corresponding compiled member to run in general case. Getting the source location back is also tricky and may break in slightly more complicated cases.

It's much simpler with testing frameworks that use attributes for each test: the attribute says explicitly where to locate the test so it's possible to build this mapping quite easily, given that compiled names of annotated members usually don't change.

Speaking or a possible contract between Expecto and an IDE, what we can realistically do is to analyze usages of a predefined set of functions/values coming from Expecto like testCase or testList inside a top level binding/member that can be located in a compiled assembly. We'd also have to restrict ways to create the test names: string literals and constants (and their concatenations) can be calculated at design time in almost all cases while results of functions like sprintf are generally not.

A structure like the following (using Expecto functions and string literals) seems to be something that's possible to work with currently:

[<Tests>]
let sameNames =
  testList "label" [
    testCase "foo1" f1
    testCase "foo2" f2

    test "bar" {
      f3 ()
    }

    // nested lists, other cases
  ]

I see such structure as somewhat limiting to the idea of producing test cases using arbitrary functions at runtime which is one of Expecto great features. Such structure, however, allows making the discovery and mapping work reliably and is something I've seen how Expecto is usually used in practice. (Please correct me if I've seen some wrong examples.)

It's definitely possible to make an IDE execute all the difficult cases but the problem here is how to make it usable for the design time story, i.e. to locate the tests before the build/run and to find a mapping between the two worlds.

[haf]

What I was thinking was something like: run expecto with --summary, then with --filter to select the test case to run; highlight constantly named tests in the IDE with a "run with" button which is capable of identifying the full name of the test, but populate the explorer post-hoc with the actual test case names? Debugging would be running with the filter flag and attaching a debugger.

[auduchinok]

I'm also looking into how we could preserve the test tree structure on our side and to live update test UI nodes while tests run. It seems what I need to achieve it is:

• unique test ids that are not names:
  • names can be duplicated (unless we refuse to support it)
  • different parent nodes may have the same names so splitting the names with / to get parents won't work reliably
• a way to get initial test nodes from the flat tests
  • reimplementing toTestCodeList could help to hook and hack it somehow but there's no way to replace it in the framework

What if we add an id field to FlatTest and pass it to the printers instead of the test names? We could add a new printers type and deprecate the current one so it doesn't break the API right away if needed. What if we add something like toTestCodeList to the config?

Knowing more about Expecto internals can you see other ways to achieve it?

Not allowing duplicate names could help with all the issues, though. How do you think, how much supporting it is ever needed at all?

[auduchinok]

OK, I think it all can be workarounded with some special treatment for duplicate names.

[haf]

What if we add an id field to FlatTest

I wouldn't mind.

Not allowing duplicate names

Can't we just not support that in the integration? It's already a warning, I don't mind removing the support for it in a major release going forward.

[haf]

Do you want to get feedback? I have rider installed, and am obviously using Expecto.

[auduchinok]

You mean feedback on Expecto support, when it arrives, right? It's probably going to appear in a some later EAP build (we haven't shipped any for 2019.3 yet actually), I'll be eager to get it then. I'll let you know when there's anything to test available. ;)