search

tmauldin / mb-unit

Automatically exported from code.google.com/p/mb-unit
0 stars 0 forks source link

Watin tests failing with Gallio RemotingException during CruiseControl.NET build #671

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1.Include ANY test or tests that use Watin in 1 specific app's CCNET build's 
Gallio/NCover testing.
2. Start the CCNET build.

***********

What is the expected output? What do you see instead?

The expected result is a successful build. The actual result is a failed CCNET 
build with all data indicating that Gallio reported that test execution failed 
(see below fro complete info). But the test failure seems to be completely 
unrelated to any specific tests, other than that it only happens when tests 
that use Watin are included in the test suite.

The tests worked fine before (using MbUnit v2.4) but stopped working when I 
upgraded to Gallio.

***********

What version of the product are you using? On what operating system?

Gallio 3.1.397.0 on Win2003, IE8, Watin 2.0.0.1, NCover 2

***********

Please provide any additional information below.

I have recently upgraded several projects from MbUnit v2.4 to Gallio 
v3.1.397.0. Everything is working great, EXCEPT for 1 thing.

In one of my projects, if the test suite includes ANY tests that use Watin, 
then the CCNET build fails because Gallio reports that test execution failed. 
However, zero tests have actually failed.

The same tests that cause a build failure run fine on my own machine, and ran 
fine in the CCNET build prior to upgrading to Gallio. One other weird symptom 
is that I have a very similar project that uses the same testing infrastructure 
and patterns, and nearly identical Watin tests still run just fine (even in the 
CCNET builds on the same server) for that other project.

Here are the test statistics from the latest failing build:
***********
Start time: 18:00:23, 2010-06-07 
End time: 18:12:32, 2010-06-07 
Tests: 15 (28 steps) 
Results: 9 run, 9 passed, 0 failed, 0 inconclusive, 6 skipped (6 ignored) 
Duration: 729.47s 
Assertions: 3 
***********

And here is the log section of the Gallio Condensed Report for that same build:
***********
[info] Host started at 6/7/2010 6:00:39 PM.
[info] Running under CLR v2.0.50727 runtime.
[info] Host stopped at 6/7/2010 6:01:04 PM.
[info] Host process exited with code: 0
[info] Host started at 6/7/2010 6:01:19 PM.
[info] Running under CLR v2.0.50727 runtime.
[info] Host stopped at 6/7/2010 6:05:54 PM.
[info] Host process exited with code: 0
[info] Host started at 6/7/2010 6:06:10 PM.
[info] Running under CLR v2.0.50727 runtime.
[warning] Watchdog timer expired!
[warning] Could not send Ping message to the remote host service.
Details: Gallio.Runtime.Hosting.HostException: The remote host service threw an 
exception or could not be reached. ---> 
System.Runtime.Remoting.RemotingException: Failed to read from an IPC Port: The 
pipe has been ended.

Server stack trace: 
   at System.Runtime.Remoting.Channels.Ipc.IpcPort.Read(Byte[] data, Int32 offset, Int32 length)
   at System.Runtime.Remoting.Channels.Ipc.PipeStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
   at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
   at System.Runtime.Remoting.Channels.Ipc.IpcClientHandler.ReadHeaders()
   at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Gallio.Runtime.Hosting.IHostService.Ping()
   at Gallio.Runtime.Hosting.RemoteHost.ProxyHostService.PingImpl() in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Runtime\Hosting\RemoteHost.cs:line 193
   --- End of inner exception stack trace ---
   at Gallio.Runtime.Hosting.RemoteHost.ProxyHostService.PingImpl() in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Runtime\Hosting\RemoteHost.cs:line 197
   at Gallio.Runtime.Hosting.BaseHostService.Ping() in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Runtime\Hosting\BaseHostService.cs:line 41
   at Gallio.Runtime.Hosting.RemoteHost.PingTimerElapsed(Object state) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Runtime\Hosting\RemoteHost.cs:line 140
[info] Host stopped at 6/7/2010 6:12:32 PM.
[info] Host process exited with code: 0
[error] A fatal exception occurred while running tests.  Possible causes 
include invalid test runner parameters and stack overflows.
Details: Gallio.Model.ModelException: An exception occurred while invoking a 
test driver. ---> System.Runtime.Remoting.RemotingException: Failed to connect 
to an IPC Port: The system cannot find the file specified.

Server stack trace: 
   at System.Runtime.Remoting.Channels.Ipc.IpcPort.Connect(String portName, Boolean secure, TokenImpersonationLevel impersonationLevel, Int32 timeout)
   at System.Runtime.Remoting.Channels.Ipc.ConnectionCache.GetConnection(String portName, Boolean secure, TokenImpersonationLevel level, Int32 timeout)
   at System.Runtime.Remoting.Channels.Ipc.IpcClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
   at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Gallio.Model.Isolation.HostedTestIsolationContext.Shim.Shutdown()
   at Gallio.Model.Isolation.HostedTestIsolationContext.RunIsolatedTaskInHost[TIsolatedTask](HostSetup hostSetup, StatusReporter statusReporter, Object[] args) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\Isolation\HostedTestIsolationContext.cs:line 149
   at Gallio.Model.Isolation.HostedTestIsolationContext.RunIsolatedTaskImpl[TIsolatedTask](HostSetup hostSetup, StatusReporter statusReporter, Object[] args) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\Isolation\HostedTestIsolationContext.cs:line 93
   at Gallio.Model.Isolation.BaseTestIsolationContext.RunIsolatedTask[TIsolatedTask](HostSetup hostSetup, StatusReporter statusReporter, Object[] args) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\Isolation\BaseTestIsolationContext.cs:line 59
   at Gallio.Model.DotNetTestDriver.ExploreOrRunAssembly(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, RemoteMessageSink remoteMessageSink, IProgressMonitor progressMonitor, String taskName, FileInfo file) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DotNetTestDriver.cs:line 196
   at Gallio.Model.DotNetTestDriver.ExploreOrRun(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IMessageSink messageSink, IProgressMonitor progressMonitor, String taskName) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DotNetTestDriver.cs:line 163
   at Gallio.Model.DotNetTestDriver.RunImpl(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IMessageSink messageSink, IProgressMonitor progressMonitor) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DotNetTestDriver.cs:line 148
   at Gallio.Model.BaseTestDriver.Run(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IMessageSink messageSink, IProgressMonitor progressMonitor) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\BaseTestDriver.cs:line 104
   at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.<>c__DisplayClass22.<RunImpl>b__20(ITestDriver driver, IList`1 items, Int32 driverCount) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DefaultTestFrameworkManager.cs:line 449
   at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.ForEachDriver[T](MultiMap`2 testFrameworkPartitions, Func`4 func) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DefaultTestFrameworkManager.cs:line 511
   --- End of inner exception stack trace ---
   at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.ForEachDriver[T](MultiMap`2 testFrameworkPartitions, Func`4 func) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DefaultTestFrameworkManager.cs:line 517
   at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.RunImpl(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IMessageSink messageSink, IProgressMonitor progressMonitor) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\DefaultTestFrameworkManager.cs:line 444
   at Gallio.Model.BaseTestDriver.Run(ITestIsolationContext testIsolationContext, TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IMessageSink messageSink, IProgressMonitor progressMonitor) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Model\BaseTestDriver.cs:line 104
   at Gallio.Runner.DefaultTestRunner.Run(TestPackage testPackage, TestExplorationOptions testExplorationOptions, TestExecutionOptions testExecutionOptions, IProgressMonitor progressMonitor) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Runner\DefaultTestRunner.cs:line 314
***********

The errors and log info appear to be completely independent of the actual tests 
I run, other than that they only happen for tests that use Watin.

Also, I have verified that all security and user settings (e.g. in the file 
system, IIS, CCNET windows services, etc.) are identical between the app where 
Watin works, and the one where it doesn't.

And rebooting the CCNET server hasn't fixed or changed any Gallio behavior for 
either app's CCNET build.

Has anyone else seen this kind of error? Or do you know how to either fix it or 
find out what the root cause is (and why it is only affecting one of the apps)?

Original issue reported on code.google.com by justin.w...@gmail.com on 8 Jun 2010 at 9:53

GoogleCodeExporter commented 9 years ago 
NOTE: My issue looks somewhat similar to Issue #557, but not exactly the same.

I do run CCNET as a service, and the exceptions look similar.

However, I am not writing anything to the Console stream in any of my code (I'm 
not sure about whetehr Watin, etc. are though).

And I am definitely not using .NET impersonation. The closest thing I am doing 
to impersonation is running the CCNET service using my own domain account as 
the service identity/user.

Original comment by justin.w...@gmail.com on 8 Jun 2010 at 9:58

GoogleCodeExporter commented 9 years ago 
Have you tried running the tests in-process? e.g. IsolatedAppDomain.

Original comment by grahamr...@gmail.com on 16 Jun 2010 at 4:10

GoogleCodeExporter commented 9 years ago 
Update:
Using the following in my scripts, I was able to successfully execute the Watin 
tests that previously caused failure:
<gallio runner-type="IsolatedAppDomain" ... />
<ncover profiledProcessModule="nant.exe" ... />

However, running the tests is EXTREMELY slow now. I also tried using these 
settings, and got virtually identical slow execution:
<gallio runner-type="IsolatedAppDomain" ... />
<ncover profiledProcessModule="" ... />

In fact, for the test project that contains the Watin tests, most of the test 
execution time is unnaccounted for by the tests themselves. For both of the 
above-mentioned test runs, I got results similar to this:

Entire UI test project:
Results: 6 run, 6 passed, 0 failed, 0 inconclusive, 0 skipped 
Duration: 485.47s 
Assertions: 15 

RemindersFixture class (total of four tests): Duration: 63.12s, Assertions: 3.
DefaultFixture class (contains 2 tests): Duration: 422.30s, Assertions: 12.
-DefaultFixture Test#1: AssemblyTest_SmokeTest: Duration: 86.92s, Assertions: 2.
-DefaultFixture Test#2: BasicPageSmokeTest: Duration: 2.30s, Assertions: 10.

As you can see, the tests in the DefaultFixture only took about 90s to run, but 
the fixture itself is listed as having taken 422s, which is very worrisome 
since I have almost all of my tests and fixtures disabled right now while I was 
debugging the Gallio issue.

I also attempted to use these settings, but they resulted in test failures of 
certain non-Watin tests which prevented the execution of any of the Watin tests 
(due to DependsOn attributes), so I didn't attempt to  (apparently caused by 
failure to read test config settings from the test projects' app.config files):
<gallio runner-type="Local" ... />
<ncover profiledProcessModule="" ... />

Is this extremely slow run speed problem a known side effect (or consequence) 
of using the IsolatedAppDomain? Are there any possible solutions or 
workarounds, or even ways to figure out what is causing the unaccounted for 
execution time?

Thanks.

Original comment by justin.w...@gmail.com on 17 Jun 2010 at 1:11

GoogleCodeExporter commented 9 years ago 
Thanks for following up with your fix.  I was having issues with an exception 
in the MSTestAdapter failing on pinging a host (I have ICMP blocked). Setting 
the MSBuild task to a runner type of "IsolatedAppDomain" resolved the issue.

My target now includes this:

        <Gallio RunnerType="IsolatedAppDomain" Verbosity="Debug"
            Files="@(TestAssemblies)"
            IgnoreFailures="true" 
            ReportDirectory="Reports"
            ReportTypes="html"> 
            <Output TaskParameter="ExitCode" PropertyName="ExitCode"/>
        </Gallio>
    <Error Text="Tests execution failed" Condition="'$(ExitCode)' != 0" />

Original comment by james.sc...@gmail.com on 8 Mar 2012 at 6:06