NUnit console test case fails when run in suite but passes individually

[neerajmathur]

NUnit console test case fails when run in suite but passes individually.

• Method of installation : zip Nunit-Console: 3.11.1

Failing Command Line D:\temp\NUnit.Console-3.11.1\bin\net35>nunit3-console.exe --workers=1 --trace=debug --labels=Before --agents=1 D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.Instance.UnitTests\bin\Release\Incite.Instance.UnitTests.dll --inprocess

53) Error : Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_send_a_fax_With_pdf_AttachmentFiles_KEY2823 System.TypeInitializationException : The type initializer for 'Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1' threw an exception. ----> System.Configuration.ConfigurationErrorsException : Configuration system failed to initialize ----> System.Runtime.Serialization.SerializationException : Type is not resolved for member 'log4net.Util.PropertiesDictionary,log4net, Version=1.2.13.0, Culture=neutral, PublicKeyToken=669e0ddf0bb1aa2a'. at Incite.ServiceBroker.Business.OutgoingFaxService.SendFax(Int64 messageId, List1 participantIds) in D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.ServiceBroker.Business\OutgoingFaxService.cs:line 43 at Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_send_a_fax_With_pdf_AttachmentFiles_KEY2823() in D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.Instance.UnitTests\Incite.ServiceBroker.Business\OutgoingFaxServiceTest.cs:line 189 --ConfigurationErrorsException at System.Configuration.ConfigurationManager.PrepareConfigSystem() at System.Configuration.ConfigurationManager.GetSection(String sectionName) at System.Configuration.ConfigurationManager.get_AppSettings() at Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase`1..cctor() in D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.ServiceBroker.Business\Common\KeystoneMessageProcessorServiceBase.cs:line 25 --SerializationException at System.AppDomain.GetHostEvidence(Type type) at System.Security.Policy.Evidence.GetHostEvidenceNoLock(Type type) at System.Security.Policy.Evidence.RawEvidenceEnumerator.MoveNext() at System.Security.Policy.Evidence.EvidenceEnumerator.MoveNext() at System.Configuration.ClientConfigPaths.GetEvidenceInfo(AppDomain appDomain, String exePath, String& typeName) at System.Configuration.ClientConfigPaths.GetTypeAndHashSuffix(AppDomain appDomain, String exePath) at System.Configuration.ClientConfigPaths..ctor(String exePath, Boolean includeUserConfig) at System.Configuration.ClientConfigPaths.GetPaths(String exePath, Boolean includeUserConfig) at System.Configuration.ClientConfigurationHost.get_HasRoamingConfig() at System.Configuration.ClientConfigurationHost.IsConfigRecordRequired(String configPath) at System.Configuration.BaseConfigurationRecord.hlNeedsChildFor(String configName) at System.Configuration.Internal.InternalConfigRoot.GetConfigRecord(String configPath) at System.Configuration.ClientConfigurationSystem.OnConfigRemoved(Object sender, InternalConfigEventArgs e)

Passing Command Line - Individual Test case D:\temp\NUnit.Console-3.11.1\bin\net35>nunit3-console.exe --workers=1 --trace=debug --labels=Before --agents=1 D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.Instance.UnitTests\bin\Release\Incite.Instance.UnitTests.dll --test=Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest

D:\temp\NUnit.Console-3.11.1\bin\net35>nunit3-console.exe --workers=1 --trace=debug --labels=Before --agents=1 D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.Instance.UnitTests\bin\Release\Incite.Instance.UnitTests.dll --test=Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest NUnit Console Runner 3.11.1 (.NET 2.0) Copyright (c) 2020 Charlie Poole, Rob Prouse Friday, 1 May 2020 12:00:34 PM

Runtime Environment OS Version: Microsoft Windows NT 6.2.9200.0 Runtime: .NET Framework CLR v4.0.30319.42000

Test Files D:\Jenkins\workspace\KeystoneCI\Phoenix\Incite.Instance.UnitTests\bin\Release\Incite.Instance.UnitTests.dll

Test Filters Test: Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest

=> Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_not_send_a_fax_if_no_fax_info 2020-05-01 12:01:01,765 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message is about to be emailed to fax service 1/05/2020 12:01:01 PM:1 2020-05-01 12:01:01,975 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - No fax enabled or no fax participants. => Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_not_send_a_fax_if_no_participantId 2020-05-01 12:01:02,144 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message is about to be emailed to fax service 1/05/2020 12:01:02 PM:1 => Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_send_a_fax 2020-05-01 12:01:02,546 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message is about to be emailed to fax service 1/05/2020 12:01:02 PM:1 2020-05-01 12:01:02,632 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message has been successfully sent to fax service at 1/05/2020 12:01:02 PM (00000000-0000-0000-0000-000000000000, MessageId: 1), ParticipantId: 1, FaxNumbers: 0288889999 => Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_send_a_fax_With_pdf_AttachmentFiles 2020-05-01 12:01:02,778 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message is about to be emailed to fax service 1/05/2020 12:01:02 PM:1 2020-05-01 12:01:02,784 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message has been successfully sent to fax service at 1/05/2020 12:01:02 PM (00000000-0000-0000-0000-000000000000, MessageId: 1), ParticipantId: 1, FaxNumbers: 0288889999 => Incite.Instance.UnitTests.Incite.ServiceBroker.Business.OutgoingFaxServiceTest.SendFax_should_send_a_fax_With_pdf_AttachmentFiles_KEY2823 2020-05-01 12:01:02,961 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase1 [(null)] [(null), (null), (null), (null), (null)] - Message is about to be emailed to fax service 1/05/2020 12:01:02 PM:1 2020-05-01 12:01:02,971 [TestRunnerThread] INFO Incite.ServiceBroker.Business.Common.KeystoneMessageProcessorServiceBase`1 [(null)] [(null), (null), (null), (null), (null)] - Message has been successfully sent to fax service at 1/05/2020 12:01:02 PM (00000000-0000-0000-0000-000000000000, MessageId: 1), ParticipantId: 1, FaxNumbers: 0288889999

Test Run Summary Overall result: Passed Test Count: 5, Passed: 5, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0 Start time: 2020-05-01 12:00:35Z End time: 2020-05-01 12:01:03Z Duration: 27.470 seconds

Results (nunit3) saved as TestResult.xml

[neerajmathur]

Attached is the TestResult.xml both passing and failing case. TestResult.zip

[ChrisMaddock]

Hi Neeraj,

I'm not sure how much we can help with this, your exception is coming from your own code, outside of NUnit. I suggest you take a look through the stack trace, and see if you can pin-point how the set up might be different between the two cases.

One thing which stands out - you have --inprocess in your first command line, and not your second. In this command line, you have both --inprocess and --agents=1 - these are two conflicting options, which will override each other in a non-defined way. (--inprocess instructs the console to run the tests in it's own process, with no agents. --agents=1 then instructs it to start up an agent process, and run the tests over there instead.)

If you don't need to run your tests in-process, try removing that flag, and see if it helps.

[neerajmathur]

Thank You! Chris, for looking into it. I am trying to find where is a problem in the code.

Surprisingly these test cases are only failing in Release mode but are passing in Debug mode. When I looked into the code there were only two places where we have used #DEBUG, I even tried removing #DEBUG code to make sure that DEBUG and RELEASE mode code will be same. Even then it is failing in release mode. Not sure what I am missing, I will continue looking into it.

[ChrisMaddock]

Thanks for the update @neerajmathur! I'm going to close this issue for now, but feel free to comment if it needs reopening later. 🙂

[neerajmathur]

Hi @ChrisMaddock , On further investigation I have found that "Optimize Code" option in Release build configuration is causing the Nunit test to fail. This flag was switched off on the debug mode that's why test cases were passing in Debug mode.

I am not sure why Optimize Code is causing an issue for Nunit to fail. .Net recommend to turn on this option in Release mode.

Do you know why Optimize Code option is causing Nunit tests to fail?

[ChrisMaddock]

Hi Neeraj - I have no idea, I'm afraid! 🙂

This isn't a common problem in NUnit test assemblies - NUnit's own tests run in release mode, without problem.

The stack trace you have doesn't include any NUnit code. I think it might be worth looking into those libraries, and trying to work out how their initialisation may differ in debug/release modes.