@Rule allow to ignore Tests

[RainerW]

Is there a way to create the same behaviour then @Ignore with a rule?

This would allow me to create a custom annoationen and rule where you can mark tests which cannot be executed in a given Environment.

[digulla]

I have a similar problem. In my case, I have fast and slow tests. Slow tests build a database, fill it with data, etc. Those take several seconds or even minutes to run.

Right now, I'm using a custom runner to enable/disable them with a System property but a rule would be much better.

So please add a property to FrameworkMethod called "ignore". When the instance is created, set this to true if an @Ignore annotation is present.

Add the usual getters and setters.

In BlockJUnit4ClassRunner and similar places, replace "method.getAnnotation(Ignore.class)" with "method.isIgnore()".

This would allow MethodRule and other JUnit extensions to flag a test method as ignored.

As it is, I can ignore a test method in a MethodRule by returning an empty Statement but in the various JUnit runners, it now looks as if the method was executed successfully which isn't correct.

[dsaff]

We intend for tests that throw AssumptionFailedException to be marked as ignored by runners. Does that work for your case?

[digulla]

At least, it's a workaround. I can move my checks into a helper method so the tests only needs a single line of code (DRY principle).

I'd prefer an annotation, though, because that would allow to easily create a report from the source. Can I throw that exception from a Statement.evaluate(), too?

[dsaff]

Yes, you can, so your assumptions can be bound up into a Rule.

[RainerW]

I use the Assumption workaround for some time now. The main drawback is that the decision if the Test is ignored, happens when already code is getting executed.

E.g. I have a selenium server rule, which starts a selenium server. Which is not the fastes operation. when now my rule detects i'm running in a non local Environment it will skip all upload tests. Sadly even in this cases the server will start.

So having statement.isIgnored() called before the actual execution, would same plenty of time.

[digulla]

Am 24.10.2011 20:05, schrieb RainerW:

I use the Assumption workaround for some time now. The main drawback is that the decision if the Test is ignored, happens when already code is getting executed.

E.g. I have a selenium server rule, which starts a selenium server. Which is not the fastes operation. when now my rule detects i'm running in a non local Environment it will skip all upload tests. Sadly even in this cases the server will start.

So having statement.isIgnored() called before the actual execution, would same plenty of time.

Try to throw the assumption in @Before or even @BeforeClass, i.e. in the code that starts the server. Should work.

Regards,

Aaron "Optimizer" Digulla a.k.a. Philmann Dark "It's not the universe that's limited, it's our imagination. Follow me and I'll show you something beyond the limits." http://blog.pdark.de/

[RainerW]

the execution order is:

• beforeClass ( here i cannot decide on a testmethod basis, only on a testclass level )
• rules[] -> before() ( this is where selenium starts )
• before ( this is where i detect the wrong environment )
• test + afters

So throwing the Assumtion in @before doesn't do the trick, it's already too late. But the environment detection is a rule anyway. So i could define the rules in a given order to circumvent this.But still i think it would be nice to skip tests with a method and not with an Exception.

[dsaff]

RainerW, have you tried the Categories support? Will that do what you want?

[RainerW]

I tried Categories, this implies that the test Executor (human Buildsystem) allready knows what can be executed. in the Selenium Sample i would like to have the logic via certain Annotations at the test. When they evaluat to X the test get's ignored.

A extreme form of this is ignoring Tests with open tickets, only Tests with a @IssueID("fixed:JIRA-123") are getting executed. This would lead to a massive set of categories.

Btw. because most "Viewers" show an Assumtion as "Green" this the assumtion solutions is only a bad workaround. This is why i now use my own runner for better feedback.

( Personally i do not use Categories; because: a) it needs a runner oder b) need to pass a start parameter and c) i hate to have interfaces just lying around )

[digulla]

Works for me. You can close this.

[Tibor17]

@RainerW , @digulla I am convinced you will not need any extra Rule to create. In the issue #142 you can ignore/exclude tests

• by creating own Filter and run your suite TestSuite and control it by build process as first alternative CategoryFilter filter= CategoryFilter.categoryFilter(null, ANY, Collections.singleton(B.class, C.class), ANY) Request request= Request.aClass(TestSuite.class); Result result= new JUnitCore().run(request.filterWith(filter));
• or as the second alternative would be to use system properties -Dorg.junit.categories.excluded=/B.java,/C.java

So your suite would be as follows:

@RunWith(Categories.class) //NULL represents categories without limitation to included. @IncludeCategory(value = null, assignableTo = Selection.ANY) @SuiteClasses({A.class, B.class, C.class}) public class TestSuite {}

It's enough to annotate every test class by itself, e.g.:

@Category(A.class) public class A {...@Test...}

[RainerW]

I think this discussen got a litte bit of track.

My original question was how could a Rule create the same behaviour as @Ignore.

The suggestet Solutions so far:

a) throw AssumtionError in your rule

drawback 1 : i see a difference between Ignore and a Assumtion. -> Ignore, means ignore, Assumtion means not today/or not on this operation-system

drawback 2 : Rules just use a decorator pattern. Even when i throw a AssumtionError in my rule's code, i cannot be sure that NO other rule had been execute before me and therefore allready done their before-test code.

b)throw AassumtionError in @Before

drawback : this makes the delegate concept of Rules useless, so this is not a solution.

c) use Categories

drawback 1: This moves the logic away form my rule into categories. ( I share Rules between projects, but not Categories! )

drawback 2: I need to use the Categories Runner ( Not compatible with 3rd party runners, eg. SpringTestRunner or some other runners i need for performance measurements)

idea 1

A possible solution would be an additional Method in the MethodRule/TestRule interface:

/* @return true if the TestMethod is actually ignored / boolean isTestIgnored( FrameworkMethod method, Object target );

This method should then be called by the Runner before any apply method get's executet. Meaning there would be a kind of additional "Test"-Phase :

setupTestMethods() // -> query all rules for ignore methods etc. before() // same as now, execute @before code of rules & test-classes execute() // execute Test after() // same as now, execute @after code of rules & test-classes

idea 2

Create an additional concept, similar to Rules, which can "Extend" Runners. This Extension can then inject logic to ignore TestMethods. And I guess adding an additional Test-Call could also be a nice feature which would be fit in such 'extension-point'.

idea 1+2

Move the isTestIgnored() Method from idea #1 into a interface. Runners now can sill query all Rules in the test-setup-phase for additional ignore-testmethods, but only Rules which actually use this would implement this interface.

[digulla]

Thanks for the great summary, RainerW. Just one point though: Making sure that the ignore rules get executed first might not be a good idea - the ignore rule might need information from other rules.

[RainerW]

This should only happen if you have closed coupled rules, which i cannot think of a scenary at all.

But execution of MethodRule::apply() and Statment::evaluate() are separated, so if this is really a problem, you need to initialize that part inside MethodRule::apply().

[RainerW]

Hm, this actually solves the problem, doesn't it ?

https://gist.github.com/2991540

I just need to throw the AssumtionException inside apply() instead of my Statement implementation. Still all Toolings will mark this as green, and not ignored, but you could argue this is a tooling issue not a JUnit issue.

[djh82]

Issue #25 deals with failed assumptions being treated as success; but only in junit and in particular the text runner. You're right about other tools such as eclipse though.

[RainerW]

Actually are there any tool that handles this correctly? e.g. Jenkins doesn't

[djh82]

I've not found one. My guess is they didn't realise they had to implement the method because they extend a n abstract class which has default no op methods. Probably didn't realise it had changed between versions.

[RainerW]

REF : suggested a solution on #25 ( https://github.com/KentBeck/junit/issues/25#issuecomment-6719074 )

[digulla]

I still prefer my solution (see this comment: https://github.com/junit-team/junit/issues/116#issuecomment-1808192) because it would be simple to add, clearly communicate intent (I want to ignore - failed assumptions are a different matter) and make the API more clean.

[dsaff]

@digulla, sorry to have left this issue hanging. If I understand, your proposal is to make FrameworkMethod mutable, with an "ignored" bit that can be turned off and on by other framework elements?

[digulla]

Yes. Alternatively, allow me to inject a custom FrameworkMethod so I could add this myself.

Or allow me to hook into the code which discovers and processes @Ignore so I can process additional annotations there.

As it is, all paths for a function like that are blocked.

[dsaff]

The intent is that failed assumptions would be treated like ignores.

In some future release of JUnit, I may figure out a mechanism for allowing test writers and running frameworks to agree on arbitrary categories of results for display. Some people want ignore and failed assumptions the same, some don't. Some want a separate "skipped" category. In general, I'm not sure JUnit as a framework wants to be in the business of arbitrating One True Vision for how people should categorize their test results. But I digress.

The problem with a mutable FrameworkMethod is that the check for method.isIgnored is before any rules are run (because we want to skip the work that rules do, as well).

[tnine]

Hey guys, Just as a workaround, I've discovered this trick with Junit 4.11. In your rule, you can use Assume to mark as ignore. I have this rule, which works when my AWS creds are missing during tests that require them.

/**
 * Created in an attempt to mark no aws cred tests as ignored.  Blocked by this issue
 * https://github.com/junit-team/junit/issues/116
 *
 * Until then, simply marks as passed, which is a bit dangerous
 */
public class NoAWSCredsRule implements TestRule {

    public Statement apply( final Statement base, final Description description ) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {

                try {
                    base.evaluate();
                }
                catch ( Throwable t ) {

                    if ( !isMissingCredsException( t ) ) {
                        throw t;
                    }

                    //do this so our test gets marked as ignored.  Not pretty, but it works
                    Assume.assumeTrue( false );

                }
            }
        };
    }

    private boolean isMissingCredsException( final Throwable t ) {
       //check here
    }
}