Assembly fails to load during Unit Test when run in VS Mac only

[LiohAu]

Hello,

I am currently facing an issue with a test project that is based on SpecFlow + NUnit 3. Basically during any test, SpecFlow tries to load an assembly and fails to load it (looks like the Assembly.Load call is failing in mono). When I try to run the same tests using NUnit console runner OR in Rider (JetBrains .NET IDE), everything works correctly.

So I looked at the MonoDevelop source code, and I think the issue may starts in NUnitProjectTestSuite.cs. It may be related to the GetCustomConsoleRunner method but unfortunately, I can't find what happens after with the GuiUnit.exe command. My guess is that some parameters may be missing here (maybe some env variables or a parameters missing ?).

Mono version : 6.10.0.104 VS Mac version : 8.6.5

PS : I am posting here and not on VS Mac, because I am sure the issue is in the MonoDevelop part.

[mrward]

In VS Mac 8.7 we added a workaround where you could specify assemblies to be loaded by the built-in test runner to solve some assembly load failures.

 <ItemGroup>
    <TestRunnerSupportAssembly Include="lib\Xwt.dll" Visible="False" />
  </ItemGroup>

[LiohAu]

Just tried in my current VS version, I does not work. I will try to update to 8.7.

Could you explain why assembly fails to load only in MonoDevelop (I am just curious) ?

[mrward]

For your particular SpecFlow problem I do not know what would be causing the assembly resolution failure. The change made in VS Mac 8.7 was made to solve some problems with the integrated runner when running the VS Mac tests since they do not local copy all the assemblies. It may be a workaround for your problem.

[LiohAu]

Ok so after updating to VS 8.7 and adding your workaround the assemblies loaded by SpecFlow are correctly loaded.

I checked, and those binaries are in the /bin/Debug/ or /bin/Release/ folder after compiling. So the issue seems to be that the mono Assembly.Load() call that SpecFlow does, is not searching in the same directories when running from VS Mac/MonoDevelop compared to Rider (or using a command line like mono nunit3-console.exe test.dll). Really would like to understand what happens behind the scenes :(

[LiohAu]

I answered too fast. Issue is solved for .dll only. Now it's the app.config which is not loaded anymore. Should I understand from this sentence the integrated runner when running the VS Mac tests since they do not local copy all the assemblies. that the tests are running from a specific directory in which any non-copied file will be missing (that could explain that app.config issue) ?

PS : after upgrading to that VS preview version, autocompletion / code indentation seems to be broken, do you know if it is a known issue of this release ?

[mrward]

That the tests are running from a specific directory in which any non-copied file will be missing (that could explain that app.config issue) ?

Possibly. Not sure how the integrated test runner works with an app.config file.

PS : after upgrading to that VS preview version, autocompletion / code indentation seems to be broken, do you know if it is a known issue of this release ?

Not that I am aware of.

[LiohAu]

Few questions again so I can try to explore this issue more deeply :

Since the app.config is correctly loaded using NUnit.ConsoleRunner.exe, Rider or VS on windows, the app.config issue is necessarily related to that test runner.

1/ Is it GuiUnit.exe that you call the "built-in test runner" ?

2/ If I understand correctly that TestRunnerSupportAssembly flag, is asking to copy the .dll in the appropriate directory so it is available for the test runner ? If true, is there another flag that I could use to also copy the app.config in the same place ? 3/ I could not find any search result on google and GitHub for that flag, it comes from closed source / non-documented code ? 4/ Do you know if I can have some logs from that test runner, maybe by setting MONO_LOG_LEVEL or another flag somewhere in the settings pane ?

[mrward]

1) There are a few test runners 2) No. TestRunnerSupportAssembly just registers the assembly with the test runner so it is pre-loaded. 3) TestRunnerSupportAssembly is not available in the open source code. This logic was around here. 4) Not sure you can without modifying the test runner code.

[LiohAu]

I can't see the runners list (permissions issue I guess), but looking at the open source code (if it is the one used by VS Mac), it's GuiUnit.exe which is launched, but I can't tell exactly what's the command is. Is it different in the closed source code ? Because the output pane does not give the command which is launched by the IDE, it only displays the output where the first line is : Loaded assembly: /Applications/Visual Studio.app/Contents/Resources/lib/monodevelop/AddIns/MonoDevelop.UnitTesting/NUnit3/NUnitRunner.exe.

Isn't there a way to get the command ran by the IDE, so I can use the mono debug flags ?

[mrward]

Sorry, wrong link. Updated it now.

[mrward]

Not sure you can get the command run by the IDE in any output. As for mono debug flags I do not think you can set them just for the unit test runner. Possibly with a custom execution mode (right clicking the test in the Unit Tests window and using Run Tests with) you can set some mono environment variables. Or run the entire IDE from the Terminal. But passing extra arguments to Mono is probably not going to be possible.

[LiohAu]

Finally passing the env var from the project settings pane worked. So I have this log : Mono: Config attempting to parse: '/Users/.../Automation/AutomationMobile/bin/Debug/AutomationMobile.dll.config'. showing that my config file is parsed, but ConfigurationManager.AppSettings.AllKeys remains empty ...

[LiohAu]

Just tried with that flag : TestRunnerCommand in the .csproj

And I had the following output

Loaded assembly: /Users/.../.nuget/packages/nunit.consolerunner/3.11.1/tools/nunit3-console.exe
Loaded assembly: /Users/.../.nuget/packages/nunit.consolerunner/3.11.1/tools/nunit.engine.api.dll
Loaded assembly: /Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
NUnit Console Runner 3.11.1 (.NET 2.0)
Copyright (c) 2020 Charlie Poole, Rob Prouse
mercredi 8 juillet 2020 10:55:39

Invalid argument: -xml=/var/folders/3w/7x22kb8s5n109rj8vdvnbhnh0000gn/T/tmp59a4ee54.tmp
Invalid argument: -run=AutomationMobile.Features.RecommendationFeature.RecommendationCall2

So I guess the only arguments of the "default test runner", are "xml" and "run", I can't see what I could add so it gets my app.config :( And looking at GuiUnit.exe help output which does not have "xml" and "run" arguments, I deduce that it's not the binary being called by VS Mac...

[LiohAu]

I still don't know why but even after updating to VS 8.7.1 the App.config is not loaded correctly.

I used the following class that I use in place of ConfigurationManager to fix the issue

public class AppConfiguration
    {
        public static AppConfiguration AppSettings = new AppConfiguration();

        public string this[string name]
        {
            get
            {
                try // for ios
                {
                    if (ConfigurationManager.AppSettings.AllKeys.FirstOrDefault(k => k == name) == null)
                    {
                        throw new Exception("AppSetting:'" + name + "' is missing in configuration file under app setting");
                    }

                    return ConfigurationManager.AppSettings[name];
                }
                catch (Exception e) 
                {
                    try
                    {
                        ExeConfigurationFileMap configFileMap = new ExeConfigurationFileMap();
                        configFileMap.ExeConfigFilename = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location).Replace("AutomationFramework", "AutomationMobile"), "App.config");
                        System.Configuration.Configuration config = ConfigurationManager.OpenMappedExeConfiguration(configFileMap, ConfigurationUserLevel.None);

                        FieldInfo configSystemField = typeof(ConfigurationManager).GetField("configSystem", BindingFlags.NonPublic | BindingFlags.Static);
                        object configSystem = configSystemField.GetValue(null);
                        FieldInfo cfgField = configSystem.GetType().GetField("cfg", BindingFlags.Instance | BindingFlags.NonPublic);
                        cfgField.SetValue(configSystem, config);

                        if (ConfigurationManager.AppSettings != null && ConfigurationManager.AppSettings.AllKeys.Contains(name))
                            return ConfigurationManager.AppSettings[name];
                    }
                    catch (Exception ex)
                    {
                        throw new Exception(message: "AppSetting:'" + name + $"'is missing in configuration file.");
                    }
                    throw new Exception(message: "AppSetting:'" + name + $"'is missing in configuration file.");
                }

            }
        }        
    }