TeamCity extension doesn't handle parallel tests

[GoogleCodeExporter]

TeamCity extension reports an incorrect number of executed tests when the 
tests are marked as [Parallelizable].

The flowId attribute has be to added to the message ( 
http://www.jetbrains.net/confluence/display/TCD4/Build+Script+Interaction+w
ith+TeamCity#BuildScriptInteractionwithTeamCity-MessageFlowId ) so that 
TeamCity can correctly group all messages related to a fixture/test.

Original issue reported on code.google.com by mauricio...@gmail.com on 17 May 2009 at 4:58

Attachments:

• teamcity.patch

[GoogleCodeExporter]

Oh!  I wasn't even aware of the "flowId" property.  Nifty.

Original comment by jeff.br...@gmail.com on 17 May 2009 at 11:07

• Changed state: Accepted
• Added labels: Type-Defect, Priority-Medium, Milestone-3.0.6, Component-Gallio

[GoogleCodeExporter]

Applied with a few modifications.
Thanks!

I'm still not sure whether the flowId will work correctly in all circumstances. 
 For
example, what will happen to our ability to report test suites?

Original comment by jeff.br...@gmail.com on 17 May 2009 at 11:02

[GoogleCodeExporter]

Now that you mention this, I see it doesn't recognize the test assembly.

3.0.6.776 without [Parallelizable] : 
http://teamcity.codebetter.com/viewLog.html?
buildId=2486&buildTypeId=bt58&tab=testsInfo&guest=1

3.0.6 with my patch: 
http://teamcity.codebetter.com/viewLog.html?buildId=2495&buildTypeId=bt58&tab=te
stsInfo&guest=1

Plus, one of the tests reports bogus timing  (7s when it should be ~540ms)

I'll try trunk with the applied patch.

Original comment by mauricio...@gmail.com on 18 May 2009 at 12:48

[GoogleCodeExporter]

The latest v3.0.6 and v3.0.7 builds are likely to give the same output.

I fear that flowId won't solve our problems.  It is intended to be used to 
report
results from completely independent processes whereas in our case we have 
branching
processes that nest.

Original comment by jeff.br...@gmail.com on 18 May 2009 at 1:43

[GoogleCodeExporter]

Here is what I am going to do.

I will modify the TeamCityExtension to produce a serial ordering of the test 
results
in the output instead of using flowId for that purpose.  What this means is that
sometimes we will delay presenting test information in order to preserve the 
correct
nesting structure.

Original comment by jeff.br...@gmail.com on 18 May 2009 at 2:03

[GoogleCodeExporter]

OK, but IMHO the message ordering should be handled by TeamCity. Otherwise the 
test timings can't be trusted. 
I'll check with JetBrains about the usage of flowId, there isn't much 
documentation about this.

Original comment by mauricio...@gmail.com on 18 May 2009 at 4:37

[GoogleCodeExporter]

We explicitly tell TeamCity what the test timings were by including a duration 
attribute.

In any case, the flowId mechanism seems to be intended to tell apart results 
from
parallel non-interacting processes whereas we have more interesting interactions
going on here.  For example, fixtures are represented as test suites in the 
output
but if we give the fixtures and their tests different flowId's then TeamCity 
will not
know they are related.

I have just applied a fix in revision 1795 (trunk) and 1796 (v3.0.6).

Original comment by jeff.br...@gmail.com on 18 May 2009 at 4:58

[GoogleCodeExporter]

Right, forgot about the duration attribute. I'll give it a try... thanks!

Original comment by mauricio...@gmail.com on 18 May 2009 at 1:22

[GoogleCodeExporter]

How does it work?

Original comment by jeff.br...@gmail.com on 18 Jul 2009 at 5:02

[GoogleCodeExporter]

I have just upgraded to 3.1.238 and it works beautifully :-)
There still seems to be a minor problem with the timings though. Compare this 
test report http://bit.ly/Uv0wg using 3.1.238 with the previous one 
http://bit.ly/WQNoc using 
3.0.6.787. Even the 3.0.6 timing for the DocumentBoost test (6s) is very 
suspicious, it 
only takes 0.8s or less when run with ReSharper.

Original comment by mauricio...@gmail.com on 20 Jul 2009 at 4:56

[GoogleCodeExporter]

I just ran the test with Icarus and it reports 8s, so it's not the teamcity 
extension's 
fault.
Not sure why... most of the tests are using Rhino Mocks, which uses Castle 
DynamicProxy, maybe something in there is blocking?

Original comment by mauricio...@gmail.com on 20 Jul 2009 at 5:04

[GoogleCodeExporter]

Here's a crazy suggestion.

Try wrapping your test code up in a new thread.  Right now MbUnit tends to 
create
very deep stacks which causes problems for some tools like the debugger.  I'd be
curious to see if Rhino.Mocks performance is also affected.  (Shortening stacks 
is
out of scope for v3.1 but it is one of my goals for v3.2.)

Original comment by jeff.br...@gmail.com on 20 Jul 2009 at 8:19

[GoogleCodeExporter]

Didn't seem to make much difference...
teamcity test log: http://bit.ly/QYy8x
changeset: http://code.google.com/p/solrnet/source/detail?r=424
On Icarus it still says 8s when running all tests. When running that test alone 
it's only 0.5s or so.
Let me know if you want me to test something else.
Anyway this isn't the original issue anymore... maybe this should be moved to a 
new, lower priority issue?
Also, how can we be sure that this is really a Gallio issue and not something 
in Rhino.Mocks or my tests or my code?

Original comment by mauricio...@gmail.com on 22 Jul 2009 at 2:33

[GoogleCodeExporter]

As you suggested, I'm going to mark this issue fixed.

It is sometimes difficult to isolate the specific cause of performance issues 
like
this because there are many and varied subtle interactions involved.  I'm glad 
it's
not another stack trace depth related issue but that would have at least given 
us a
tidy hypothesis to pursue.  Feel free to open another issue to follow up on this
separately in v3.2.  I hope the performance is acceptable for the time being.

Incidentally, you might take a look at the Gallio.Framework.Tasks static class. 
 It
provides helpers for tests that spawn threads and processes.  ;-)

Original comment by jeff.br...@gmail.com on 22 Jul 2009 at 5:16

• Changed state: Fixed