Inherited TestFixtureAttribute causes Gallio to Crash ReSharper to Crash Visual Studio

[GoogleCodeExporter]

[Originally sent to JetBrains]

We've actually narrowed the problem down to the Gallio test runner, which
unfortunately is causing ReSharper to crash Visual Studio.  The scenario is
as follows:

Using ReSharper 4.1 (or 4.0) and VS2008.

As part of utilizing BDD with the Gallio test framework, we have an
attribute called Concern - ConcernAttribute.  All of our TestFixture's
inherit from a base "ContextSpecification".  Originally we had all our
tests in the same project as our little test framework. 
ContextSpecification was decorated with TestFixture, and each actual
specification was decorated with ConcernAttribute and inherited from
ContextSpecification - leading to every specification being decorated with
TestFixture.

The issue came when we decided to move the tests into the same project as
our production code, then use msbuild to conditionally compile our test
code so it doesn't go out with the production system.  In order to support
this we moved our test framework out to another project "TestFramework". 
At this point, the Gallio test runner could not find the tests within the
project, it seems that Gallio is unable to handle attribute inheritance
from outside of the current project.  In order to fix this, we changed the
ConcernAttribute to inherit directly from TestFixtureAttribute.  This
seemed to work pretty well until ReSharper started falling over sideways
and crashing Visual Studio.

In order to reproduce the problem on your machine, open the project,
disable ReSharper via Add-in manager and then re-enable ReSharper.

Original issue reported on code.google.com by dmorgant...@gmail.com on 3 Feb 2009 at 5:01

Attachments:

• CrushingIssue_Gallio_and_R#.zip

[GoogleCodeExporter]

Yikes.

Original comment by jeff.br...@gmail.com on 4 Mar 2009 at 6:19

[GoogleCodeExporter]

I'm not sure this is still an issue but I will look into it for v3.0.6 Update 2.

Original comment by jeff.br...@gmail.com on 27 Apr 2009 at 6:01

• Changed state: Accepted
• Added labels: Component-ResharperRunner, Milestone-3.0.6, Priority-High, Type-Defect

[GoogleCodeExporter]

Still an issue.

Original comment by Feldman.Sean.Work@gmail.com on 27 Apr 2009 at 2:07

[GoogleCodeExporter]

I cannot reproduce this with v3.0.6 Update 1 and ReSharper v4.5 using your 
sample
project.  Are you sure it's still an issue here?

There was a bug a while back where we had unbounded recursion that would kill 
VS but
I think it was fixed.

Original comment by jeff.br...@gmail.com on 28 Apr 2009 at 3:43

[GoogleCodeExporter]

will look into this to see if with 4.5 it's not happening.

Original comment by Feldman.Sean.Work@gmail.com on 28 Apr 2009 at 5:39

[GoogleCodeExporter]

Ok, I tried it with R# 4.5 and and the latest Gallio/MbUnit.
This time VS.NET does not crush. What does happen, is R# complains about the 
fact
that it gets an exception while exploring tests. VS compiles the project
successfully, but R# reports an error. I used TestDriven.NET to run tests, and 
it
does run, despite the fact that its reported that no test exists. I used the 
original
project attached to the thread to do my tests. R# had "Solution Compilation" 
turned on.
Thank you,
Sean

Original comment by Feldman.Sean.Work@gmail.com on 28 Apr 2009 at 3:25

Attachments:

• 2009-04-28_Resharper_and_vs.net_2008_testing_framework_issue.swf

[GoogleCodeExporter]

In the bundle you sent me, the Gallio.dll and MbUnit.dll in the source tree 
were old
v3.0.5 versions.  Be sure that you upgrade these dlls to v3.0.6 otherwise you 
will
experience various version compatibility errors such as the 
InvalidCastException seen
in your screencast.

BTW, thanks very much for producing the little screencast.  It was very helpful!

P.S.  I noticed TDNet is running in "AdHoc" mode because Gallio failed to run 
the
test itself.  You might notice other issues from that (like setup won't run).  
That
will go away when the other problem is taken care of.

Original comment by jeff.br...@gmail.com on 28 Apr 2009 at 6:32

[GoogleCodeExporter]

Jeff, 
thanks a lot for the advise - I completely skipped the fact it was an older 
version.
Looks better with 3.0.6, but still some issues. Video attached.
Sean

Original comment by Feldman.Sean.Work@gmail.com on 29 Apr 2009 at 4:12

Attachments:

• 2009-04-28_TestFixture_is_ok_Observation_is_still_not.swf

[GoogleCodeExporter]

Can you try this version:
http://ccnet.gallio.org/Distributables/GallioBundle-3.0.6.773-Setup-x86.msi

or if 64bit

http://ccnet.gallio.org/Distributables/GallioBundle-3.0.6.773-Setup-x64.msi

Thanks.

Original comment by jeff.br...@gmail.com on 29 Apr 2009 at 4:56

[GoogleCodeExporter]

Jeff,
same thing - sorry.

Original comment by Feldman.Sean.Work@gmail.com on 29 Apr 2009 at 8:00

[GoogleCodeExporter]

Note sure about the TDNet issue.  I just tried this:

    public class ConcernAttribute : TestFixtureAttribute
    {
    }

    public class ObservationAttribute : TestAttribute
    {
    }

    [Concern]
    public class Spec
    {
        [Observation]
        public void Test()
        {
        }
    }

Works in TDNet.  The ReSharper issue is more complex.

Original comment by jeff.br...@gmail.com on 15 May 2009 at 2:14

[GoogleCodeExporter]

Now as for ReSharper, the trick is that it needs to be able to resolve the 
assembly
that defines your custom framework attributes.

To confirm this, I moved those two attributes up there into a different 
assembly and
copied it next to MbUnit.dll in the Gallio bin folder.  Then like magic R# 
could show
lightbulbs for those test types.

(Just for kicks I called the assembly MbSpec.  ;-)

The trick here is that the new framework assembly then becomes loaded into 
Visual
Studio so it cannot be modified without shutting down Visual Studio.

I'm thinking about adding a new custom attribute of some kind to make it 
possible to
specify these custom framework assemblies or other plugins alongside your tests 
such
that they don't need to be installed into the Gallio bin folder itself.  The 
caveat
is that they would end up locked in memory as soon as they were used for the 
first time.

Original comment by jeff.br...@gmail.com on 15 May 2009 at 2:22

[GoogleCodeExporter]

I should mention that the red strike issue was fixed separately 
and will be in the next build.

On reflection I will hold off on adding a test assembly based 
mechanism for registering plugins since it will add a lot of 
complexity whereas we could instead work to make the plugin 
discovery and installation process better in general.

Original comment by jeff.br...@gmail.com on 15 May 2009 at 2:32

• Changed state: Fixed