carlpett / xUnit-TeamCity

A TeamCity plugin for running .NET xUnit tests
Apache License 2.0
45 stars 12 forks source link

runner is not working in TeamCity 9.1 #1

Closed aizumoff closed 8 years ago

aizumoff commented 9 years ago

Hi, Carl! I've incorporated your runner into TC9.1 (just released) and noticed that it's not working (build log for this step is just empty and finishes within 1sec).

It seems to me it worked in TC9.0 (we've migrated to 9.1 just a week ago).

Do you have an idea why it stopped working?

I really love your plugin as it provides convenient setup in TC as opposed to other workaround runners via MSBuild&etc.

Thx!

apostlethwaite commented 9 years ago

I'm seeing the same on Team City 9.1.1 (build 37059).

9swampy commented 9 years ago

Did anyone work out what's wrong or find an alternative that works? We're postponing updating TeamCity just for this runner... FWIW it works perfect on 9.0.1 (build 32116)

tstewart65 commented 9 years ago

I installed this on TeamCity 9.1 today and also the problem with it not running at first. I tracked done my issue to the path I was putting in "Assemblies containing tests". I finally got it to run by using "bin/*/.Tests.dll", without the quotes. This gives me all of the assemblies that end it ".Tests.dll" under the bin folder.

wwwlicious commented 9 years ago

Not sure if it's what you want but I've created a meta-runner for dotCover and xunit that works on my instance of 9.1.2 and supports traits, usual "works on my machine* caveats apply!

carlpett commented 9 years ago

Very sorry for not replying - for some reason I did not get notifications on issues being created. I'll have a look at 9.1 compatibility during coming week.

carlpett commented 9 years ago

@tstewart65: Glad you got it working. The path supports the same "ant-style" wildcards as most things in TeamCity, so you can also do something like **/*.Tests.dll to get any depth recursion.

carlpett commented 9 years ago

@aizumoff @apostlethwaite and @9swampy: I upgraded my local install to 9.1.3 (build 37176), and at least my first test worked out. What versions of xUnit are you using? Could you supply some log files?

9swampy commented 9 years ago

@carlpett Just to be clear, we're working perfectly fine in 9.0.1 using xUnit 2 (and appreciate the work you put in thus far), was just looking for confirmations that it will continue to work should we upgrade before doing so following on from @aizumoff comment.

SeanKilleen commented 9 years ago

EDIT:

Nevermind, this one is on us -- the build had changed to not build the whole solution. If your test projects aren't built, there probably won't be a DLL for xUnit to find. :)

Original below for posterity.


@carlpett Experiencing this in TeamCity 9.1.3:

image

I'm set up to do all *.Tests.dll projects:

01_xunitconfig

For example, I have tests at [check-out root]\src\test\Common.Tests\bin\Release\Common.Tests.dll

When I run, I see no test output on the build:

02_notests

And I see nothing under the console output when executing the MSBuild tests:

03_nooutput


Let me know if I can help provide any other information -- thanks!

apostlethwaite commented 9 years ago

Hi, @carlpett I'm seeing the same output as @SeanKilleen has posted a screenshot of. The step in the build log has no output and no tests appear to have been logged.

I've set my params to: Version 1.9.2 Assemblies: %teamcity.build.checkoutDir%\depot1\Test\APITests\bin\x64\Release\APITests.dll

I've verified the APITests.dll exists on the agent after a build. Tried both x64 and x86 flavours.

We use two different versions of xunit in different projects with the same results. 1.9.0.1556 and 1.8.0.1545

You could say I'm taking a gamble that 1.9.2 works with older versions, but seeing as there's no output, I can't tell if that's the problem or not at this point.

Anything else I can provide? Cheers, alex

carlpett commented 9 years ago

@apostlethwaite Thanks for the info! Could you try running it the test manually in a console? <agent home dir>\tools\xunit-runner\bin\1.9.2\xunit.console.exe <path to assembly> /teamcity. Just to make sure that the xunit runner itself can actually find the tests.

apostlethwaite commented 9 years ago

@carlpett Yeah that appears to work. When I log on to the agent, I get:

C:\Users\TCAgent>"C:\TeamCity\tools\xunit-runner\bin\1.9.2\xunit.console.exe" "D:\TeamCity\work\e39c853fcd1bee\depot1\Test\APITests\bin\x64\Release\APITests.dll" /teamcity
xUnit.net console test runner (64-bit .NET 4.0.30319.34209)
Copyright (C) 2013 Outercurve Foundation.

xunit.dll:     Version 1.8.0.1545
Test assembly: D:\TeamCity\work\e39c853fcd1bee\depot1\Test\APITests\bin\x64\Release\APITests.dll

##teamcity[testSuiteStarted name='APITests.dll']
##teamcity[testStarted name='APITests.TestOne']
##teamcity[testFinished name='APITests.TestOne' duration='4475']

and so on... so that appears to be okay.

I guess the problem is this:

Could not load file or assembly 'APITests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An attempt was made to load a program with an incorrect format.

Which is the message I get when I try to load an x86 dll. Perhaps my memory is incorrect, and what actually happened is we tried the x86 flavour of our job, we got no output, and then never enabled that step for x64.

I'll try the x64 one and get back to you on that.

It would great if you could bundle both runners and decide on the arch (or if it's lots of work to decide on demand, make it a parameter on the build step page).

carlpett commented 9 years ago

Aha, that would explain it. Multi-arch support is tracked in issue #4. I'll have a look if it is easy to determine the arch, but otherwise it'll have to be a dropdown.

Please try the x64 one and see if it works!

apostlethwaite commented 9 years ago

Please try the x64 one and see if it works!

Hi Calle, Unfortunately it doesn't. I get the same in the build log, no output. I've run procmon on the agent whilst the job runs, and I can't see any instances of the xunit runner being loaded.

I've got another step in the job, like:

set xunit_console="%teamcity.build.checkoutDir%\depot1\ThirdParty\xUnit\xunit.console.clr4.exe"

%%xunit_console%% "%teamcity.build.checkoutDir%\depot1\Test\APITests\bin\x64\Release\APITests.dll"

And that works okay, so I know that the project can run on this agent.

Any further ideas?

mnak commented 9 years ago

The same problem :( In the build log file on the server there is nothing

carlpett commented 9 years ago

@apostlethwaite, @mnak Thanks for your patience. I've created a debug build with additional logging, as well as fixing some logging that was broken in the base builder class (Java is not my language of choice...). Could you try to install this version and see if you get any more information? 1.0.1-debug: https://dl.dropboxusercontent.com/u/43785918/xUnit.zip

apostlethwaite commented 9 years ago

@carlpett Sorry for the delay - had to be at work to try this.

I got:

[09:42:50]Step 8/12: Run tests (xUnit)
[09:42:50][Step 8/12] [XUnit DEBUG] - Constructed build process
[09:42:50][Step 8/12] [XUnit DEBUG] - Build process started
[09:42:50][Step 8/12] Starting test runner at C:\TeamCity\tools\xunit-runner\bin\1.9.2\xunit.console.exe
[09:42:50][Step 8/12] [XUnit DEBUG] - Build process was finished

in my build log, but no tests appear to be run.

SeanKilleen commented 9 years ago

Jumping back in to see if I can help out.

It does seem that there are no test assemblies found by the xunit runner.

How are you telling the plugin which assemblies to find? For example, in ours:

image

carlpett commented 9 years ago

@apostlethwaite No worries. That looks a bit strange, it should at least have printed the header from the runner. Could you have a look in the windows event log to see if it reports something there? Probably something like .NET Application as source, in case it is a crash.

spib commented 8 years ago

I'm having the same issue. I can see the xunit runner plugin on the build agent (under tools) and executing that runner manually against my test dll on the agent works fine. Looking in the teamcity log files there's nothing obvious about why it's not running, I just get nothing in the log files for that step. Any ideas?

carlpett commented 8 years ago

Hi, sorry to hear that! Could you tell me a few things about your setup?

spib commented 8 years ago

Thanks for the quick reply

carlpett commented 8 years ago

@spib I'm unable to reproduce, unfortunately :( My current best guess is that it has to do with the stdout redirection that doesn't work under some circumstances (likely related to my Java skills...). Could you try to create a "test" that writes a file to disk somewhere? That way we'll know if the test runner actually gets invoked.

apostlethwaite commented 8 years ago

@carlpett Hi, Sorry it's been such a long delay, had vacation for a while.

I'm still seeing the same kind of thing in my output:

[Step 3/3] [XUnit DEBUG] - Constructed build process
[16:03:11][Step 3/3] [XUnit DEBUG] - Build process started
[16:03:11][Step 3/3] Starting test runner at C:\TeamCity\tools\xunit-runner\bin\2.0.0\xunit.console.exe
[16:03:11][Step 3/3] [XUnit DEBUG] - Build process was finished

with no tests being run.

As last time, if I log on to the agent and run:

C:\work>c:\TeamCity\tools\xunit-runner\bin\2.0.0\xunit.console.exe D:\TeamCity\work\e39c853fcd1bee\depot1\Test\APITests\bin\x64\Release\APITests.dll

It runs the tests as expected.

Could you have a look in the windows event log to see if it reports something there

I've checked the event viewer and there were no crash events reported at the time the job ran.

We've updated to TeamCity 9.1.6 (build 37459) running on Windows Server 2008 R2 since I first commented on this issue.

Thanks, alex

apostlethwaite commented 8 years ago

@carlpett Hi again.

I've downloaded your prerelease1-1, but still having no luck.

My build log just contains:

[16:30:00]Step 4/5: xUnit
[16:30:00][Step 4/5] Runner parameters { Version = 2.1.0, runtime = .NET 4.5, platform = AnyCPU/MSIL}
[16:30:00][Step 4/5] Starting test runner at C:\TeamCity\tools\xunit-runner\bin\2.1.0\xunit.console.exe
[16:30:01]Step 5/5: New build step (xUnit)
[16:30:01][Step 5/5] Runner parameters { Version = 2.1.0, runtime = .NET 4.5, platform = x86}
[16:30:01][Step 5/5] Starting test runner at C:\TeamCity\tools\xunit-runner\bin\2.1.0\xunit.console.x86.exe

Thanks, alex

carlpett commented 8 years ago

@apostlethwaite Thanks for reporting back! Did you have opportunity to test my suggestion above of creating a test which writes something to disk? That would help determining if it is indeed unable to spawn a process, or just having some issues with output redirection.

If that still gives the same results, I'll spin up another environment and try to get it as close as possible to yours - any customizations from a blank 2008R2? English locale or some other language? Anything else that comes to mind?

evilB commented 8 years ago

I have the same issue, I've done as you said and created a test generating a file to see if it's run. The file is created when run manually, however from teamcity the file doesn't get created, suggesting a file selection issue.

I have the issue on two build agents: 1 windows server 2012 r2 1 windows 8.1

carlpett commented 8 years ago

@evilB Could you post a build log? It should output the results of the file selection, among other things. Are you running the 1.1 release, or 1.0?

evilB commented 8 years ago

I'm running the 1.1.0 release.

Build log (doesn't indicate the things you mention, perhaps it is a different log?): [16:55:26]Step 4/4: Run xUnit tests (xUnit) [16:55:27][Step 4/4] Runner parameters { Version = 2.0.0, runtime = .NET 4.5, platform = AnyCPU/MSIL} [16:55:27][Step 4/4] Starting test runner at C:\TeamCity\BuildAgent\tools\xunit-runner\bin\2.0.0\xunit.console.exe

Assembly files include list Sarspace.Services.Tests\bin\Debug\Sarspace.Services.Tests.dll **\bin***Tests*.dll

apostlethwaite commented 8 years ago

HI @carlpett .

I built your plugin and have done some experiments.

I was getting no results back from the findFiles method

for(File assembly : finder.findFiles(context.getWorkingDirectory())) {

If I change my assembly list from depot1\Test\APITests\bin\x64\Release\APITests.dll to depot1/Test/APITests/bin/x64/Release/APITests.dll It starts working (i.e. use forward slashes).

It's not obvious to me that that is a requirement, but maybe that's because I'm not familiar with 'ant-like patterns' or whatever the TC docs call them.

Having a look at the FXCop plugin though, it seems that they convert '\' to '/' before using the ANTFileFinder (see splitFileWildcards in http://svn.jetbrains.org/teamcity/plugins/fxcop/trunk/fxcop-agent/src/jetbrains/buildServer/fxcop/agent/FxCopBuildService.java). Maybe that could be done?

alex

carlpett commented 8 years ago

@apostlethwaite Thanks, that is some great detective work! Must say I've never thought about it either, I just routinely use forward slashes, but assumed it worked with backslashes as well. Fixing this, and also adding a warning that no assemblies were matched (which really should have been there from the start)

carlpett commented 8 years ago

@apostlethwaite @evilB I have released version 1.1.1, which both detects and warns about no matching assemblies, as well as converting \ to / so that the pattern matcher will work with both cases. I hope this will fix your issues! Again, big thanks to @apostlethwaite for finding the cause.

apostlethwaite commented 8 years ago

@carlpett Cheers Calle, I've deployed 1.1.1 and that's working for me.

evilB commented 8 years ago

I can confirm it works for me too. Thank you for the effort!

carlpett commented 8 years ago

Excellent, thank you all for the patience in resolving this!