feat(testrunner): test discovery using vstest native code

[GustavEikaas]

Description

Uses vstest and F# instead of dotnet test -t for test discovery, this fixes a lot of bugs and brings a lot of enhancements.

Features:

• [x] Fixes MSTest discovery issues
• [x] Better lookup on testresults using Id instead of fully qualified name
• [x] F# scripts will update automatically when installing new versions of easy-dotnet.nvim
• [x] Automatically resolve sdk path

Issues:

• [x] Requires an absolute vstest.console path to work (ships with dotnet sdk confirmed)
• [x] Test on linux
• [x] Replace old test parser with new programatically
• [x] Value too large (partition json with F#)
• [x] Update readme (how to find sdk path etc..)
• [x] Add check if name doesnt contain . and use namespace instead (or merge name and namespace) (MSTest specific)

[franroa]

Hi, I could test this if you are ready for feedback

[GustavEikaas]

Yeah knock yourself out, you need to add path to config like this

{
  test_runner = {
    vstest_path = "C:/etc..."
  }
}

On windows i had to open visual studio command prompt and write where vstest.console.exe to get the path.

[GustavEikaas]

Ah maybe it wasnt so ready for feedback as I was thinking, keeps crashing on bigger projects. The test_parser.fsx file in the vim.fn.stdpath("data")/easy-dotnet/test-parser.fsx needs to be deleted so this branch can recreate it

[franroa]

Was looking into vstest on linux. I came to this docu: https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-vstest Have you tried this?

[GustavEikaas]

Yeah i made a question on my issue on the microsoft github asking whether the path is necessary or if i can reference something preinstalled in dotnet cli

[franroa]

Ok, cool. I will wait for the answer of the vstest project. I got some troubles trying to install vstest on linux. In the worst case I could use wine, but this is for me the last resort.

To be honest, actually I am using WSL, so theoretically I could use the vstest executable of windows, but this is always to avoid due to the performance issues.

In any case, I could give a try to both options

[GustavEikaas]

If this comes down to either WSL or wine for linux compability, I will probably discard this PR and improve stdout parsing, or research other alternatives

[GustavEikaas]

Seems like it ships with the dotnet sdk. I just tested with vstest_path = "C:/Program Files/dotnet/sdk/8.0.400/vstest.console.dll" Can you test if you find it and if it works on linux @franroa

[franroa]

I can confirm that there is a file there: /usr/lib/dotnet/sdk/8.0.108/vstest.console.dll

Will test with that path later today. Let me know when this is ready

[franroa]

Just tried only out of curiosity. It works pretty well, but there is an issue with (apparently) json objects when the project is too big:

Vim:E1510: Value too large: 
stack traceback:
    [C]: in function 'json_decode'
    .../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:191: in function <.../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:185>
Vim:E1510: Value too large: 
stack traceback:
    [C]: in function 'json_decode'
    .../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:191: in function <.../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:185> function: builtin#18 Vim:E1510: Value too large: 
stack traceback:
    [C]: in function 'json_decode'
    .../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:191: in function <.../easy-dotnet.nvim/lua/easy-dotnet/test-runner/runner.lua:185>

[GustavEikaas]

Yeah Im aware of this, I was unsure when this would be a problem, ill just have to update the F# script to partition the json output

[GustavEikaas]

Either dump as partitioned json in stdout or write to os.tmpname and cleanup afterwards

EDIT: Why not just print one json object per line in stdout and just read stdout line by line, parsing each and putting it into a list??

[franroa]

per line reading seems to be a good solution. I will test this branch afterwards

[GustavEikaas]

@franroa Should work now, I added the json partitioning using line by line as mentioned above. Let me know if there are any more issues

EDIT: Seems like some tests in discovery might be printing with \n once in a while. Getting some malformed json errors but rerunning fixes it

Seems like I have to write to a json temp file and read line by line if i want a solid solution on this

[franroa]

It works pretty well in general. By every new run I get an error by two or three cases. The cases don't seem to be always the same. This is an output:

Malformed json: Vim:E474: Expected string end: {"Id":"13800776-ddd7-f32f-b72b-27c134ed2059","Namespace":"xxxxTest.Controllers.Calendars.CalendarQueryControllerTest.GetCalendarById","Name":"xxxxx.Test.Controllers.Calendars.CalendarQueryControllerTest.GetCalendarById(commandHandlerError: True)","

The json object is not parsed to the end, or the error is not showing the whole output

[GustavEikaas]

Yeah ill try writing the results from F# execution to a file and read line by line. I suspect there is some autowrap magic somewhere when printing long lines using dotnet fsi

[GustavEikaas]

I fixed it by emitting to a tmpfile and reading line by line from that to avoid accidental linebreaks. cant reproduce your issue no more. btw the repo is internal/private so i cant see it

[franroa]

Sorry send the wrong link

[GustavEikaas]

Finally! The specflow example works with full namespace support 🤘

[franroa]

This really really cool! Awesome how you got it working. Will test as soon I can

[GustavEikaas]

Using https://www.nuget.org/packages/Microsoft.TestPlatform.TranslationLayer it outputs the full namespaces even on MSTests. You can also press g on any test to go to file and line!

So the autojumping can work now

[franroa]

Just tested. Very well done! Just one issue. This stuck on Running: EDIT: This is only happening when I choose that one test. When I run as a group it works! EDIT2: When running a group, all tests are always marked as passed, and the classData tests with arguments stuck on Running

[GustavEikaas]

Okay so you open the InsertCalendarTest.xml file which indicates that this should be one singular unit test. But in your xml it specifies multiple <UnitTestResult .../> nodes with the same id(!?). Does all the nodes have the same status? I clearly see why its failing, I just have no idea what is so special about the InsertCalendarTest.

Below is what the xml should have looked like, with a single <UnitTestResult /> node

Can you try and make a super basic repro where you get the same behaviour as the InsertCalendarTest. aka when you run it the xml contains multiple duplicate entries

[franroa]

Yes I will do it. I need to analyze myself why this is showing up as a normal test and not as classdata. I have many tests like that, and there is no difference between those and the "normal" classdata tests

[GustavEikaas]

Just to confirm, using dotnet test -t it outputs a single entry for InsertCalendarTest?

[GustavEikaas]

@franroa Is the bug you're experiencing a blocker for merging this do you think?

[franroa]

I think you can merge, and I add a bug for this. Not sure if I will manage to put together a repro today, so I would'nt want to block

[franroa]

Hi, just got to know that I have to move to other project, so I will not be using neovim for now. If you still need a repro I can try to do it, otherwise if you don't need it we can merge this PR

[GustavEikaas]

Oh wow! Good luck on your new project! Dont worry about the repro, someone will likely run into the issue down the road anyways.

[franroa]

ok! thanks! You are doing a great job here!