Open charlesroddie opened 2 years ago
@baronfel I'm taking up your suggestion to try Expecto (via https://github.com/YoloDev/YoloDev.Expecto.TestSdk) and it's good so far but I'm stuck at this point.
Even "Rectangular hierarchy", in which folders of degree 0 contain only tests, and folders of degree n contain folders of degree n-1, and the main degree is 3, doesn't work:
There seems to be a big problem connecting Expecto to Test Explorer. Test Explorer may be hacked together with strings but it definitely allows for deeper hierarchies than this.
Am I right in posting here rather than in https://github.com/YoloDev/YoloDev.Expecto.TestSdk ?
Hi.
It looks like you're using Visual Studio.
I don't use Visual Studio very much anymore, but back in the day I used to use adamchester/expecto-adapter. It looks like it's been archived since then (I guess people use YoloDev.Expecto.TestSdk for that now), but I remember getting the nesting in the VS Test Explorer to work fine. See screenshot in https://github.com/adamchester/expecto-adapter/pull/47. You could try seeing if it still works but it might need an older version of VS.
Usually I just use the terminal for running tests now (ftestCase
and ftestList
are useful) so don't know much about YoloDev.Expecto.TestSdk, but I tried your example in VSCode to see what it's like with Ionide.
The nesting works fine for what you had in your "Rectangular hierarchy".
"Some other test" doesn't even show up in the right place though, it should be nested under "List".
Also, it only seems to identify testCase
s with a function defined in-line (CaseAAA1 shows up but CaseAAB1 doesn't). And of course, tests generated in a for loop don't show up either.
There seems to be an issue with it Ionide or VSCode though. When I try to run tests from the test explorer it just hangs indefinitely showing the spinner. Nothing shows up in any of the VSCode output logs.
The
TestResults/Ionide.trx
file is generated successfully and has all the expected tests and test names, so I suspect if it actually ran properly it might show all the tests and in the correct spots.
The output of dotnet test -t
is exactly what I'd expect it to be, as is the Ionide.trx output file, so I'd guess the issue is probably in the interaction between YoloDev.Expecto.TestSdk and Visual Studio. (And the Ionide/VSCode issue is probably unrelated)
The following Tests are available:
List.Some other test
List.ListA.ListAA.ListAAA.CaseAAA1
List.ListA.ListAA.ListAAA.CaseAAA2
List.ListA.ListAA.ListAAB.CaseAAB1
List.ListA.ListAA.ListAAB.CaseAAB2
List.ListA.ListAB.ListABA-generated-list.CaseABA0
List.ListA.ListAB.ListABA-generated-list.CaseABA1
List.ListA.ListAB.ListABA-generated-list.CaseABA2
List.ListA.ListAB.ListABA-generated-list.CaseABA3
List.ListA.ListAB.ListABA-generated-list.CaseABA4
List.ListA.ListAB.ListABB.CaseABB1
List.ListA.ListAB.ListABB.CaseABB2
...
Note that the VSCode feature is in its infancy, so I'd err on the side of a bug in our test discovery and execution. This issue should focus purely on the discovery and execution of expecto tests via the Yolodev adapter, which provides the bridge to the 'dotnet test' world that powers the CLI and VS experiences.
How does the VSCode feature discover and execute tests then (and where is it implemented?) if it's not with the yolodev adapter? I thought it runs something like dotnet test --logger "trx;Ionide.trx"
to generate the report, meaning it would be using the same "'dotnet test' world" as VS.
But agreed, this issue should focus on the Yolodev adapter with VS
The discovery happens via syntax analysis, which is way faster but not as accurate. The execution should happen via dotnet test, however. Can you raise an issue about any test execution issues on the Ionide VSCode repo?
Given tests in a folder structure:
Is there a way to have Test Explorer match this?
Project, Namespace, Class isn't right![image](https://user-images.githubusercontent.com/19760720/164944918-f194b609-2719-4a7e-8cf1-973956a1b2d7.png)
Project, Class isn't right either![image](https://user-images.githubusercontent.com/19760720/164944933-898c8bf1-8728-4b43-8b14-e270ac003040.png)
@cmeeren in https://github.com/haf/expecto/issues/364 was getting some correspondence, but the samples there are very constrained - every test is exactly 2 levels deep in the hierarchy.
Is there a spec for the Test Explorer behaviour, or should I figure out by trial and error what Expecto structures result in what outputs and then try to make a set of consrained types (FolderLevel1, FolderLevel2) and then try to generate an altered Expecto folder structure to get the desired Text Explorer behaviour?