search

nunit / teamcity-event-listener

NUnit Engine extension that helps integration with teamcity
MIT License
11 stars 10 forks source link

Running multiple nunit console runner processes #62

Closed Sharken03 closed 5 years ago

Sharken03 commented 5 years ago

The situation is that we have a TeamCity installation, where we start multiple NUnit 3 Console Runner in order to reduce time to test assemblies. TeamCity service messages are enabled through the --teamcity parameter and the test assemblies are built against the NUnit 2 Framework.

In order to separate live output from the runners, TeamCity uses the FlowId. The problem is with EventConverter2.cs where all output has the same FlowId. I have therefore made a change, where FlowId is controlled by the environment variable TEAMCITY_PROCESS_FLOW_ID like this (PowerShell example): $env:TEAMCITY_PROCESS_FLOW_ID=1;nunit3-console.exe Tests.dll --teamcity

That enables TeamCity to distinguish between runners, thus showing correct live test ouput. I have a working code change, that i could provide as well.

NikolayPianikov commented 5 years ago

@Sharken03 Thanks for info. I've reproduced it. I found simple solution. Will be fixed soon

NikolayPianikov commented 5 years ago

@Sharken03 try using this package

Sharken03 commented 5 years ago

@NikolayPianikov Thank you, i have tried the package referred and as far as NUnit 2 goes, i would say it works perfectly.

I am a bit concerned that NUnit 3 could cause problems, as i do not see a way of controlling Flowid using TEAMCITY_PROCESS_FLOW_ID. But if a problem should appear, i will be open a new issue.

NikolayPianikov commented 5 years ago

@Sharken03 NUnit 3 should work as expected here. The root flow id is received from an environment variable and it is used here. Also there is a test for this case. Anyway feel free to create the issue

Sharken03 commented 5 years ago

@NikolayPianikov Not sure if this warrants reopening the issue, but something is behaving different when using FlowId. The same test now shows up twice in TeamCity and it is related to how the NUnit Console output is parsed by TeamCity.

The test has not been moved from the test assembly where it resides. But the suite name is different for the test in two different builds. One build uses two elements from the namespace of the test, while the other build uses three elements from the namespace. If the suite name could be changed to the test assembly name, then it would be a lot more consistent. But perhaps that is an issue within TeamCity ?

The issue seems to be this one: https://youtrack.jetbrains.com/issue/TW-54170

NikolayPianikov commented 5 years ago

@Sharken03 To check this issue you could run tests from the command line with the arg --teamcity and check tests' names. But anyway, is it possible to share a simple solution on github reproducing this issue? What version of TeamCity is installed?

Sharken03 commented 5 years ago

The TeamCity version is 2018.2.2.

I have tried to run with --teamcity and am seeing ##teamcity[testSuiteStarted name='X.Y' flowId='1.1'], where X.Y is the start of the test namespace. For other test suites i see ##teamcity[testSuiteStarted name='some_assembly.dll' flowId='1.1'].

It could be related to the naming of our test assemblies, we use a mix of either underscores or dots, such as 'my_test_assembly.dll' and 'another.testassembly.dll'. It appears that this affects the testsuite name being output, which in turn leads to inconsistent test suite naming. A test suite name without '.dll' or '.exe' seems to be a bug.

The assemblyname from https://github.com/nunit/teamcity-event-listener/blob/b0831113bec722aa0ad64ae7b66ce02a5f4f56e8/src/extension/ServiceMessageFactory.cs#L41 is the problem, as this is not always the correct assemblyname for the test.

NikolayPianikov commented 5 years ago

@Sharken03 You're right. I will try to understand where it is better to fix, I mean nunit extension or in the TeamCity core.

Sharken03 commented 5 years ago

That sounds great, if something needs to be tested just let me know.

NikolayPianikov commented 5 years ago

@Sharken03 Is it possible to share current stdOut for case 'X.Y' and for case 'some_assembly.dll' with the arg --teamcity? I do not want to miss any details and I'd prefer to create some regression tests on these cases. You could use my email to sent info. Thank you in advance.

Sharken03 commented 5 years ago

@NikolayPianikov I might be able to put something together, i guess you would need both result file output and output from using --teamcity. Is there a secure way to send the logfiles ?

NikolayPianikov commented 5 years ago

@Sharken03 just attach files to this ticket in the comment visible to JetBrains team only

Sharken03 commented 5 years ago

@NikolayPianikov Have attached sample output now.