Add or document a way to write unit tests for Godot C# projects (e.g. with nunit3)

[dsge]

Describe the project you are working on:

A 3D spaceship/spacebase building game (think Space Engineers)

Describe the problem or limitation you are having in your project:

The more code you write to do custom calculations the more cumbersome it becomes to keep looking back to see if previously written code still functions as expected. Adding more code or refactoring the current code is also much easier if you can build on properly (unit)tested foundations.

As an actual example from my current project, I have to calculate how fast my spaceship can accelerate/decelerate given:

• the current power level of the spaceship
• thrusters power consumption
• number of thrusters
• thrusters maximum performance in a given direction
• ...and possibly more factors

Manually testing all of these again and again (and again) is possible, however it's time consuming and inefficient.

Describe how this feature / enhancement will help you overcome this problem or limitation:

Proper unittests can save a lot of time by quickly telling me if my (new or old) code still functions as expected, thus saving me time that can be instead spent on new features.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

While having a GUI is nice and all, I think CLI support is the more critical for this particular feature. Running the tests locally and in CI can eliminate further bugs while also allowing for better cooperation on the project with other contibutors.

Workflow in case it is possible to run tests directly using nunit3-console.exe:

• (build the game dll using Godot editor)
• ./path/to/nunit3-console.exe /path/to/game.dll --where "<filter for testcases>" --pause
  • the where option allows to filter for testcases (for when you only want to run specific ones instead of all of them)
  • the pause option allows to attach external debuggers to debug test cases
• nunit3-console.exe outputs the tests results and returns 0 if all tests passed (for CI support)

Workflow in case the tests need to be ran through the Godot editor's CLI:

• (build the game using Godot editor)
• ./path/to/Godot_v3.2-stable_yourplatform --path "/path/to/game" --run-nunit3-console --where "<filter for testcases>" --pause
  • the where option allows to filter for testcases (for when you only want to run specific ones instead of all of them)
  • the pause option allows to attach external debuggers to debug test cases
• Godot_v3.2-stable_yourplatform outputs the tests results provided by nunit3-console.exe and returns 0 if all tests passed (for CI support)

Describe implementation detail for your proposal (in code), if possible:

Unfortunately I only have limited understanding on Godot's source code or how c# (mono) support is implemented, so I do not know what needs to be implemented or changed to allow this (if anything). If using nunit3 is already possible, then this feature request is actually a request for documentation/examples.

What I do know however is that right now if you try to run your game dll with nunit3-console.exe your testcases will actually work until your testcase reaches the first call to Godot (more specifically to anything inside GodotSharp.dll). You will be bombarded with error messages like this:

cant resolve internal call to "Godot.NativeCalls::godot_icall_Camera_Ctor" (tested without signature also)

Your mono runtime and class libraries are out of sync.
The out of sync library is: /app/.mono/temp/bin/Tools/GodotSharp.dll

When you update one from git you need to update, compile and install
the other too.
Do not report this as a bug unless you're sure you have updated correctly:
you probably have a broken mono install.
If you see other errors or faults after this message they are probably related
and you need to fix your mono install first.

What this error message actually means (despite what it says) is that the c# function called Godot.NativeCalls::godot_icall_Camera_Ctor is defined as an extern function in GodotSharp.dll but no actual implementation is found anywhere. The same code (same project) does work as expected when exporting/running it through Godot Editor.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

Not that I am aware of.

Is there a reason why this should be core and not an add-on in the asset library?:

I think at least documentation on this should be included within Godot's official documentation, even if any implementation is outside of the main source code. The reason for this is to avoid confusion by providing a singular official way for running c# unittests.

[van800]

Normally tests are executed directly in TestRunner process. This causes any unmanaged calls to Godot api-s to fail. So if you want to test a piece of logic without calling unmanaged Godot api-s it would work. Basic example https://github.com/van800/SkyOfSteel/commit/6515c3fdd508083950d57300f1f6bf25e08a0025

I have seen 2 different implementations, which overcome this limitation:

1. TestRunner has communication channel with Application via network and tells it to load some assembly and execute some specific piece of code. http://dev-in-test.blogspot.com/2014/04/resharper-integration-tests.html
2. TestRunner is embedded in the Application and tests are executed in the application process. For example Unity does that. It provides API for a plugin to subscribe to future test-results and run tests. Like this one https://github.com/van800/com.unity.ide.rider/blob/master/Packages/com.unity.ide.rider/Rider/Editor/UnitTesting/RiderTestRunner.cs#L67

Both approaches are not easy/fast to be implemented.

[mattrobin]

This seems like it should be a pretty important issue. Most serious projects will probably need to have unit tests, and having the standard unit testing tools available seems essential. Even for my little hobby project in Godot, I feel quite hindered by not being able to easily use standard style C# tests. It's been making me reconsider if Godot is really the engine I should be using, or if I should be switching to something that better enables regular coding practices. I can only imagine how much this deters more serious game developers from using Godot. This is probably too extreme of a blanket statement, but typically more experienced developers are more likely to rely on unit tests. And having experienced developers choose Godot as their standard tool will certainly help Godot move forward.

[martinruefenacht]

I can only agree with @shianiawhite, this issue is single-handedly the reason why I switched from Godot to Unity. Even a temporary, before full integration, documented path would be useful to have.

[Calinou]

@martinruefenacht Please don't bump issues without contributing significant new information. Use the :+1: reaction button on the first post instead.

[evan-boissonnot]

So, I'm sorry, I don't understand the release plan about this feature ?

• How could I unit test now my godot C# project ?

  Is there a way I can do it until you implement in Godot ?

• What do you plan to do ?

  Create option in IDE (in the menu) will be great For example : stop launch and run the UI without green unit tests ?

[AlexDarigan]

So this is self-promotion but it is also very relevant to the issue at hand.

I ran into this issue recently myself when I wanted to change from GDScript to C#. I decided to wrap WAT (The Godot GDScript Plugin) in C#. You can find the reddit thread with more detailed information and links here.

This has a GUI, CLI, Parameterized Testing & a number of other features available. Since it is built mainly using GDScript (with C# on user-interfacing scripts) it doesn't run into the same errors as the more established C# Frameworks meaning you can test any Godot Class without worry.

With that said I do think it is probably a good idea to implement some form of communication to the godot api if 1) it is relatively painless to implement 2) is generic so this fix can allow Godot to work with any standard c# test framework.

[mattrobin]

There were a couple comments that involved something to do with the UI. I just wanted to clarify that (I think) the main part of this issue requires no sort of UI changes. It's just that a test runner currently can't call any Godot related code, and for the tests to be run, the runner needs to be able to make those calls. I suspect a majority of the people who would be writing tests would be using an external IDE anyway (though I certainly may be wrong).

[van800]

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

With the following changes:

1. On Rider plugin side https://github.com/JetBrains/godot-support/pull/58
2. On the game side https://github.com/van800/godot-demo-projects/pull/4 Both plugins are open-source on github, anyone can try.

Still some work ahead.

1. Make debugging work
2. Make runner.tscn an invisible commandline, if possible.
3. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq Should I create a separate proposal or we can discuss here?

[neikeq]

3. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq

Yes, that would be very welcome!

[van800]

For Rider 2021.1EAP feature was merged in Rider Godot support plugin, it still requires a small preparation step - see instructions.

[Flavelius]

@van800 It works fine for 3.2.3, but fails for 3.2.4.beta6 (using the same project), the test result is 'Inconclusive: Test not run', with GodotSharp assembly not found.

[BenMcLean]

This is flat out intolerable and is one of the issues making me think of abandoning Godot for my next project.

[Calinou]

@BenMcLean Please don't bump issues without contributing significant new information. Use the :+1: reaction button on the first post instead.

[evan-boissonnot]

Hi, what's up about the prob in C# unit test. I still get "ECall methods must be packaged into a system module." error when I want to create my own unit tests ....

[evan-boissonnot]

Have you never use Unity 3D ? There is truely way, as any C# developper can expect, to unit tests : using unit tests in VS 2019. Not with tricks or hacks ... just using Unit Tests in IDE.

That what I need in Godot mono. :/

So for me the proposal here is not good.

[AlexDarigan]

Hi, what's up about the prob in C# unit test. I still get "ECall methods must be packaged into a system module." error when I want to create my own unit tests ....

This is an issue about Assembly cross-talk. The JetBrains Rider plugin has a basic system working to handle this. Alternatively you can use WAT Mono which is a C# Unit Testing Plugin I made from the ground up to work with Godot without issue.

[GeorgeS2019]

Alternative based on xunit. Promising!

https://github.com/fledware/GodotXUnit

[mscharley]

One option that would go a long way for me would even to just be able to unit test a library with no Godot references in it but I'm struggling to get that setup in a way that feels natural to me because of the insistence from the editor about where the sln and csproj files end up - ie, at the same level as the project.godot file. That makes it harder to have a nested testing csproj or indeed even the library project itself. The ideal project layout for me would been this but it's currently impossible as far as I can see:

/project.godot
/Project.sln
/src/Project.csproj
/lib/SharedLib.csproj
/tests/Tests.csproj

It seemed like godot didn't mind too much where the solution file was, but the csproj must be at the same level as the godot project file. Without wanting to enforce my layout on others, it would be nice if we could at least tell Godot where to look for the csproject it needs to build. The build command itself only seems to rely on the solution file but I keep getting this error (3.3.rc7.mono):

--- Debugging process stopped ---
 modules/mono/mono_gd/gd_mono_utils.cpp:370 - Microsoft.Build.Exceptions.InvalidProjectFileException: The project file could not be loaded. Could not find file "C:\[path-to-game]\Game.csproj"  C:\[path-to-game]\Game.csproj
 editor/editor_node.cpp:5497 - An EditorPlugin build callback failed.

[GeorgeS2019]

@mscharley
Please refer to the proposal of .NET5 SDK project discussion.

Issues of existing c# project managed by Godot Mono Tools

• GodotXUnit: for it to works with a .NET5 SDK format, we must specify exactly which c# files to be included to avoid the main Godot project csproj in .NET5 sdk format to AVOID including all cs files within the same level and below the directory which the .csproj located. This nullify the benefits brought by the simplicity of .NET5 SDK format. It is still unclear why the author of GodotXUnit mixes both the legacy .NETFramework with the .NET5 SDK formats.
• It is still unclear how this is addressed in Godot4 mono. I am still investigating.

I do agree, we need to continue to discuss the most optimal approach for unit testing for Godot c# developers.

[van800]

I haven't tried it myself, but feels like worth sharing here: https://github.com/semickolon/NUG

[git2013vb]

My attempt is using XUnit + VSCode + C# + dll + .NET Core Test Explorer.

I tried to set xunit for a dll.

The class I'm using can't be godot classes. If I add a Godot base class I have an error https://godotforums.org/discussion/27574/xunit-net-give-error-when-test-godot-class.

The current csproj for xunit is this one:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
    <DebugType>portable</DebugType>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4"/>
    <PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="16.11.0"/>
    <PackageReference Include="xunit" Version="2.4.1"/>
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3"/>
    <PackageReference Include="xunit.runner.console" Version="2.4.1"/>
    <PackageReference Include="xunit.runner.msbuild" Version="2.4.1"/>
    <PackageReference Include="coverlet.collector" Version="3.0.2">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <Reference Include="GodotSharp">
      <HintPath>/home/vale/Downloads/godot_mono/Godot_v3.4-beta5_mono_x11_64/GodotSharp/Api/Debug/GodotSharp.dll</HintPath>
    </Reference>
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\GodotUtilities.csproj"/>
  </ItemGroup>
</Project>

When you use:dotnet new xunit it will create a basic csproj that you have to complete as I did + a setting in vscode extension like

then you will have the tree in vscode ( see first picture) For what I need it will be suffice to find a solution for the error posted in forum. Once it is compatible with Godot objects I think it will be good option to choose from to have unit tests in godot using C#.

[van800]

I have found out that not only nUnit would work from Rider, but also xUnit and probably any other unit test framework, which is supported by Rider. Feel free to check. My updated example, which contains different tests https://github.com/van800/godot-demo-projects/pull/7

[ashelleyPurdue]

If you're comfortable with including your unit tests in the same project as your game code, I've made an editor plugin that runs NUnit tests:

https://github.com/ashelleyPurdue/NURFG

[MikeSchulze]

Hi i working to to bring GdUnit3 c# support. It's current in a early alpha stage but suggestions are always welcome. Feel free to have a look at https://github.com/MikeSchulze/gdUnit3/tree/GD-129-1 (Please note it is still in progress and not yet public)

GdUnit3 is able to run c# test directly inside the Godot editor. Support to run inside VC Code and Rider will be added later.

[hhyyrylainen]

@GeorgeS2019 using visual studio 2019, is not free for all users (the community edition license has quite many restrictions on who can use it). Also I doubt many CI pipelines want to include visual studio, just to run unit test. At least I won't be, even trying, to fit that into our docker based CI pipeline for our game made with Godot engine.

[hhyyrylainen]

I don't really understand what you are saying... I just wanted to bring up that promoting visual studio as being free needs to have an asterisk attached, as it is not truly free for all users. And I also wanted to mention that for our team, being able to run the unit tests in CI environment (so from a command line, which would be also useful for decoupling the unit test running from requiring any IDE) is a requirement before we can start using it.

[MikeSchulze]

Sort update form GdUnit3 with c# test support.

I hard working to release V2.0.0 with initial C# test support, i made good progress. The new version will come with a Visual Code extension to run and navigate over your test direct inside VSC. https://www.youtube.com/watch?v=qD-1BQuWwLs https://mikeschulze.github.io/gdUnit3/

Feedback always welcome

[MikeSchulze]

Update:

With GdUnit3 v2.3.0, the C# api will be complete and depends on SDK 6.0 and can be easily bundled via Nuget packages. https://github.com/MikeSchulze/gdUnit3/milestone/28

After release i will start to publish tutorials how to work with GdUnit3 from beginners to expert.

[ncallaway]

@mscharley

I was able to achieve a similar setup, with a shared library that contains no Godot references, and xunit to unit test that library.

I used the following project structure:

gd/project.godot
gd/Project.sln
gd/Project.csproj
lib/SharedLib/SharedLib.cspoj
tests/SharedTests/SharedTests.csproj

• dotnet new classlib --framework [your-target-framework] -o lib/SharedLib
• dotnet sln gd/Project.sln add lib/SharedLib/SharedLib.csproj
• dotnet add gd/Project.csproj reference lib/SharedLib/SharedLib.csproj

At this point, I'm able to call code in the SharedLib from my C# code inside the godot project.

• dotnet new xunit --framework [test-framework] -o tests/SharedTests
• dotnet add tests/SharedTests/SharedTests.csproj reference lib/SharedLib/SharedLib.csproj
• dotnet sln gd/Project.sln add tests/SharedTests/SharedTests.csproj

At this point I can write tests that call code in the SharedLib from the SharedTests project, and run xunit tests with:

• cd tests/SharedTests
• dotnet test

I used netstandard2.1 for my Godot and shared lib target framework and net6.0 for my test framework.

[jolexxa]

If it's helpful to anyone, I have also ended up creating a customizable test execution system called GoDotTest that allows you to debug tests directly from VSCode, debug the currently open test, and run tests from the command line. I use it extensively across a number of projects to run tests for CI/CD.

[fbrier]

@MikeSchulze

With GdUnit3 v2.3.0, the C# api will be complete and depends on SDK 6.0 and can be easily bundled via Nuget packages.

I REALLY like the idea of just adding a Nuget package to a NUnit (or whatever) project. Unit testing, not spinning up the entire game engine, requires being able to instantiate Godot Objects (Node(s) or Reference(s) or whatever), or classes that derive from them. Not required, but an API that could trigger the invocation of _Ready() and _Process() methods for all instances would be helpful in writing tests. Or triggering events from a simulated keyboard, mouse or controller - ie: VirtualKeyboard.KeyPress('R').

At one point, I looked at the engine code that registers types and instantiates objects, but it was all internal and not accessible using the GDNative C/C++ API. If I am missing an existing way to do that, please let me know. If not, what if one of the engine libraries contained the class registration and script engine? A new library (not a normal part of the engine) would expose the required methods. The two libraries, plus a C# wrapper, would be bundled as a Nuget package. Then you could use any C# test framework or even C++ test framework. Such a library would make using Dependency Injection (DI), TDD, and CI trivial.

[MikeSchulze]

You can simply test your scenes in GdUnit3 https://user-images.githubusercontent.com/347037/185459511-a65373e0-505a-404e-87c2-522651c59790.mp4

[GeorgeS2019]

Can I test scene now in c# using GdUnit3?

[MikeSchulze]

@GeorgeS2019` yes there also exists a scene runner

  [TestCase(Timeout = 2000)]
        public async Task SimulateFrames()
        {
            ISceneRunner runner = ISceneRunner.Load("res://test/core/resources/scenes/TestScene.tscn", true);

            var box1 = runner.GetProperty<Godot.ColorRect>("_box1");
            // initial is white
            AssertObject(box1.Color).IsEqual(Colors.White);

            // start color cycle by invoke the function 'start_color_cycle'
            runner.Invoke("start_color_cycle");

            // we wait for 10 frames
            await runner.SimulateFrames(10);
            // after 10 frame is still white
            AssertObject(box1.Color).IsEqual(Colors.White);

            // we wait 90 more frames
            await runner.SimulateFrames(90);
            // after 100 frames the box one should be changed the color
            AssertObject(box1.Color).IsNotEqual(Colors.White);
        }

With next major release v2.3.0 the c# API will be feature complete and update the documentation

[dmlo]

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

With the following changes:

1. On Rider plugin side Run NUnit/XUnit tests in Godot process JetBrains/godot-support#58
2. On the game side Nunit van800/godot-demo-projects#4 Both plugins are open-source on github, anyone can try.

Still some work ahead.

1. Make debugging work
2. Make runner.tscn an invisible commandline, if possible.
3. Integrate EntryPoint.cs and runner.tscn into main Godot, so users wouldn't need to copy them manually inside their project.

If I make a PR which solves (3), would it be welcomed? @neikeq Should I create a separate proposal or we can discuss here?

For those coming here via google in 2023 because you want to run NUnit unit tests in Rider on Godot projects, this is the solution:

1. Install the Godot support plugin
2. Add the Runner.cs & Runner.tscn files from the above github repo to your Godot project in a folder called RiderTestRunner
3. Run your tests as normal in Rider
4. Give a star to van800 for solving your (and my) problem

Thanks @van800. This is super clean.

[a2937]

I managed to make a proof-of-concept integration Rider UnitTestRunner with Godot.

Is there an equivalent option for those of that use Visual Studio 2022?

[dioptryk]

I experimented a bit and was able to create a TestAdapter based on Microsoft.TestPlatform. I implemented ITestDiscoverer and ITestExecutor. When referenced in a Godot mono project (should work for both 3.x and 4.x, since it's netstandard2.0), it is properly recognized by Visual Studio and allows listing and executing tests.

Tests are listed based on special attribute in the main project assembly. The project also contains a .cs, that inherits a custom TestRunner class, that uses the same attribute to run tests. This is non-avoidable, I think, because we have to use Godot to actually execute stuff, so we need a script.

The executor simply runs Godot without window (version does not matter), but only once. It passes list of chosen tests as argument and also the Runner script as "-s" Godot parameter. Then the Runner does its thing and, for each test, outputs some special stuff on console, which is in turn understood by the executor as the specific test result.

This is a very simple solution that seems to work pretty well. No debugging, though, I'm not sure it's possible with this approach, but I've only spent two hours on this for now. Not sure if someone is interested in this.. ? I will try making a NuGet package in a few days (so there's no need for a Godot plugin), since I need a test solution myself. Also, I've no idea why, but dotnet test lists the tests correctly ("-t"), and yet when asked to execute them, tells me there are none. Probably some stupid issue somewhere.

[GeorgeS2019]

@dioptryk https://github.com/godotengine/godot-csharp-visualstudio/issues/48

[dmitrii1991]

https://github.com/godotengine/godot-proposals/issues/432#issuecomment-1399644644

the version Godo 4.1 does not work. the scene starts but nothing comes bac 0

[zorbathut]

@dioptryk do you have your test-framework code up anywhere? I'd love to take a look at it.

[MikeSchulze]

Just for info, I'm current working to finalize by gdUnit4 test adapter. Features:

• run all or a subset of tests
• debug your tests
• navigate to failures

Here a sneak peek

[dioptryk]

@zorbathut I worked a bit more on it, but I ultimately understood that Godot is not really something that should be used in unit test scenario. Try to have as many types as possible not inherit from GodotObject (implement composition in your class hierarchy) and unit test that.

I realized that what I really wanted are pipeline test results. And this is actually trivial, because you can generate test result xml in JUnit format with anything, be it GDScript or C#. You don't need to use XUnit, NUnit or whatever, you only need result files. So what I'm doing now is running godot --headless on the pipeline, starting some "scenario" scenes and verifying if they complete as expected. I also change the engine time multiplier for stuff to happen faster :) Then I am generating XML result file for my "scenarios" (see https://github.com/testmoapp/junitxml/tree/main) and publish it on Azure DevOps using PublishTestResults@2 task. And there you have it: the single test you see is actually a scene where Player character walks a few steps and picks up an Item. Then the scenario running code checks if the test succeeded (Player has an Item). If this single passes, it means that whole bunch of stuff works as expected (whole initialization, map loading, movement, inventory etc.)

[pgirald]

Hey!

Any progress on this issue? I'm currently exploring ways to run isolated tests on my scenes using xunit and the dotnet test command. Any solution that lets me test just the scene logic without the engine would be fantastic.

Alternatively, knowing how to run all the necessary Godot dependencies before executing dotnet test would work too. The important thing is not having that "ECall methods must ...." error when loading a scene from a test method.

Currently, I'm using Godot 3.5. Cheers!

[GeorgeS2019]

https://github.com/godotengine/godot-proposals/issues/432#issuecomment-1852363796

https://github.com/MikeSchulze/gdUnit4Mono/issues/35

using the latest gdUnit4API TestAdaptor on VS2022 from @MikeSchulze

[GeorgeS2019]

C# (Scene) Test-Driven-Development (TDD) using Godot4 in VS2022

https://youtu.be/JLI0cIYMgr0

using the latest gdUnit4API TestAdaptor from @MikeSchulze

https://github.com/godotengine/godot-proposals/assets/49812372/72bb91a1-dd1c-4467-8d07-3b0d3beadd4b

[van800]

gdUnit4 VS integration is cool and it also turned out that it is easy to use in the JetBrains Rider. It requires Rider 2024.1, Godot Support 2024.1.157 and enable VSTest in the Rider settings. I have tried it on MACOSX, but I expect it to work x-plat everywhere. There is no need to manually set the GODOT_BIN, since Rider already knows where the Godot is installed.

[a2937]

Now all we need is a stable version of GDUnit built into Godot. :P

[MikeSchulze]

Now all we need is a stable version of GDUnit built into Godot. :P

OMG, this requires a complete rewrite in c++ :) Maybe binding by the GdExtension, but this is also a lot of work to do.

[a2937]

Can't they like publicize some of the existing test runner stuff? They already have pretty good tests for the GDScript implementation itself. https://github.com/godotengine/godot/tree/master/modules/gdscript/tests

[AThousandShips]

What do you mean publicize?