Add ExclusiveResource programmatically

[U1F984]

I have several test classes, each with multiple test methods inside. Some of those classes are injected with a parameter, and when running classes in parallel those classes should not run parallel. If I understand correctly, the @ResourceLock annotation can help here, but it would be very inconvenient to manually add that annotation to hundereds of tests. Ideally a extension could take a look at the class, see that it has this parameter and somehow add the ExclusiveResource to it.

My JUnit execution parameters:

-Djunit.jupiter.execution.parallel.enabled=true
-Djunit.jupiter.execution.parallel.mode.default=same_thread
-Djunit.jupiter.execution.parallel.mode.classes.default=concurrent
-Djunit.jupiter.execution.parallel.config.strategy=fixed
-Djunit.jupiter.execution.parallel.config.fixed.parallelism=8

[marcphilipp]

We could add a new extension point for this:

interface ExclusiveResourceProvider extends Extension {
    default Set<ExclusiveResource> provideExclusiveResourcesForClass(Class<?> testClass, Set<ExclusiveResource> declaredResources) {
        return declaredResources;
    };
    default Set<ExclusiveResource> provideExclusiveResourcesForNestedClass(Class<?> nestedClass, Set<ExclusiveResource> declaredResources) {
        return declaredResources;
    };
    default Set<ExclusiveResource> provideExclusiveResourcesForMethod(Class<?> testClass, Method testMethod, Set<ExclusiveResource> declaredResources) {
        return declaredResources;
    };
}
interface ExclusiveResource {
    String getKey();
    ResourceAccessMode getAccessMode();
    static ExclusiveResource of(String key);
    static ExclusiveResource of(String key, ResourceAccessMode accessMode);
}

[jbduncan]

This feature might be quite useful for my PR on junit-pioneer to allow "resources" - arbitrary objects - to be instantiated and shared amongst tests declaratively.

Specifically, if each "resource" were to have its own ResourceLock that could be created programmatically, then it would simplify making my extension thread-safe.

The proposed API leaves me with one question, though: if my extension were to implement ResourceLockProvider, when would the provideX methods be called?

[marcphilipp]

It would have to be called immediately before starting execution on the engine level.

[jbduncan]

Okay, thank you!

I wasn't sure at first if this would work for me, since my extension's resources are created at test runtime. But I think it could work now, as all shared resources in my extension are declared with @Shared annotations, which I imagine my potential ResourceLockProvider could search for reflectively.

So I'm happy with the proposed API. 👍

[marcphilipp]

Team decision: Let's go with the above proposal.

@U1F984 @jbduncan Is one of you interested in working on this?

[jbduncan]

@marcphilipp I'd love to! However I still have https://github.com/junit-pioneer/junit-pioneer/issues/348 to finish, so I'm happy for anyone else to pick this up before me.

[jbduncan]

For clarity, my JUnit Pioneer work is not actually blocked on this feature, as I've found an alternative solution to programmatic ResourceLocks for my needs.

[jbduncan]

@marcphilipp A follow-up query from me: do you know if JUnit Jupiter already ensures that ResourceLocks are obtained in such an order as to prevent the dining philosophers problem, by any chance?

If not, then I, or whoever picks this up, will need to ensure the locks are sorted when being moved from ResourceLockProviders to the rest of the test framework, by some definition of "sorted".

[marcphilipp]

They are sorted here: https://github.com/junit-team/junit5/blob/6a47ae152003a6ac3e77f2e604159b71e4f82953/junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/LockManager.java#L61

[jbduncan]

@marcphilipp Great, thank you for answering so quickly. 👍

[jbduncan]

@marcphilipp I'd love it if you assigned this to me, as I've managed to park my junit-pioneer work and find time to start looking at this.

[marcphilipp]

@jbduncan Sorry it took 6 days, but you're now assigned.

[jbduncan]

Sorry for the silence on this! I was a bit lost looking through the JUnit 5 codebase when I looked at this issue last year. But now that my JUnit Pioneer PR that would have used this feature is finally done, I'll see if I can find the time to revisit this issue later this week, and write down anything I need help with. :)

[jbduncan]

@marcphilipp I have a few follow-up questions which I was hoping you could help me with.

For the proposed ExclusiveResourceProvider,

1. Does a provide* method's declaredResources contain just the resources found so far for the given class/method/nested class, or does it contain all resources found so far by JUnit?

2. Do the declaredResources contain resources found through @ResourceLocks?

   @Test
   @ResourceLock(value = "a")
   @ExtendWith(MyExclusiveResourceProvider.class)
   void test() {
   }

3. If so, does the ordering matter? As in, does MyExclusiveResourceProvider's declared resources contain resource "a" when running with test2, but not with test1?

   @Test
   @ExtendWith(MyExclusiveResourceProvider.class)
   @ResourceLock(value = "a")
   void test1() {
   }
   
   @Test
   @ResourceLock(value = "a")
   @ExtendWith(MyExclusiveResourceProvider.class)
   void test2() {
   }
   
   class MyExclusiveResourceProvider implements ExclusiveResourceProvider {
       @Override public Set<ExclusiveResource> provideExclusiveResourcesForMethod(Class<?> testClass, Method testMethod, Set<ExclusiveResource> declaredResources) {
           return declaredResources;
       }
   }

[jbduncan]

I've pushed my ongoing spike to https://github.com/junit-team/junit5/pull/3096 for the sake of transparency. :)

[jbduncan]

@marcphilipp I'm feeling a bit stuck now, and I was wondering if you could help me.

It looks like NodeTreeWalker is responsible for discovering tests and, most importantly, their @ResourceLocks. However, to get resource locks from ExclusiveResourceProviders, JUnit would need to discover them as extensions, which are currently discovered by NodeTestTasks, which in turn depend on the NodeTreeWalker! So there's a cycle of dependencies (maybe even a web), and I can't see how to break it, so I was was wondering if you had any thoughts?

Alternatively, would you be up to discussing this remotely some time? If so, feel free to contact me on my GitHub email address.

[jbduncan]

@marcphilipp I don't suppose you've had the time to look at my last comment recently, have you? 🙂

[marcphilipp]

@jbduncan Sorry, I had fallen quite behind in reading my GitHub notifications. 😬

It looks like NodeTreeWalker is responsible for discovering tests and, most importantly, their @ResourceLocks. However, to get resource locks from ExclusiveResourceProviders, JUnit would need to discover them as extensions, which are currently discovered by NodeTestTasks, which in turn depend on the NodeTreeWalker!

I'm not sure I fully understand the question. ExclusiveResourceProvider can't be a regular extension since it needs to be applied at discovery time, i.e. it should not extend Extension. We already have a few of this kind of extension, e.g. DisplayNameGenerator and MethodOrderer. So I think we'll need a new annotation that goes along with ExclusiveResourceProvider, maybe @ExclusiveResources(MyExclusiveResourceProvider.class)?

[jbduncan]

@jbduncan Sorry, I had fallen quite behind in reading my GitHub notifications. 😬

No worries! Likewise, sorry for not responding immediately. :)

I'm not sure I fully understand the question. ExclusiveResourceProvider can't be a regular extension since it needs to be applied at discovery time, i.e. it should not extend Extension. We already have a few of this kind of extension, e.g. DisplayNameGenerator and MethodOrderer. So I think we'll need a new annotation that goes along with ExclusiveResourceProvider, maybe @ExclusiveResources(MyExclusiveResourceProvider.class)?

@marcphilipp Oh, I hadn't considered that. That sounds like it could break the dependency cycle.

But admittedly I've lost interest in making programmatic exclusive resources a reality, so I'll leave this issue and my draft PR for someone else who in interested in picking things up.

[marcphilipp]

No worries. Thanks for letting us know!

[VladimirDmitrienko]

Hi @marcphilipp

Me and my team interested in this feature. We have a lot of tests and would like to configure locking strategy without adding / removing @ResourceLock annotation.

I've prepared a draft implementation of this functionality and would be very grateful if you take a look at it. Please, tell if you're happy with a such approach (in such case I'll finalize the PR), or you have another approach in mind. https://github.com/junit-team/junit5/pull/3889

Regards, Vladimir

[marcphilipp]

@VladimirDmitrienko Thanks for the draft PR! Could you please describe your use case in more detail? Is a class-level annotation sufficient or would you rather register such a provider "globally" for all tests?

[VladimirDmitrienko]

@marcphilipp Thanks for a quick response!

We have lots of tests which interact with our application through different levels: UI, API, DB. When we run our tests in parallel, sometimes they may "intersect" because a test or a group of tests may modify a common state used by other tests (a typical problem actually).

My initial thought was to use @ExclusiveResources(MyExclusiveResourceProvider.class) on the base test class and assign different locks right from MyExclusiveResourceProvider. In such case we would be able to add a necessary level of isolation between tests based on class / package names patterns. Making sure that particular kinds / groups of tests are never executed at the same time, but still be executed in parallel with tests which not require locks.

---

So, responding to your question, the current approach would be sufficient for our case, but I guess that a "global" ExclusiveResourceProvider may be an even better solution since it will not require users to have a "base" class for all the tests. For such purpose we could introduce a new property, maybe something like:

junit.jupiter.execution.parallel.locks.provider = org.com.example.MyExclusiveResourceProvider

---

To summarize, currently I see three options to set ExclusiveResourceProvider:

1. A class-level @ExclusiveResources annotation.

2. junit.jupiter.execution.parallel.locks.provider property.

3. Combine both approaches to be able to override the property with an annotation. Looks like the most flexible since fits both:

• small-medium size projects which can benefit from having all the logic "in one place";
• big projects with huge test classes hierarchy: each Base...Tests class can have it's own @ExclusiveResourceProvider.

So, I'm looking forward to your feedback to continue to work on whichever approach we choose.

[VladimirDmitrienko]

@marcphilipp maybe you had a chance to take a look at my last comment?

Also, I'd appreciate if you assign this issue on me 🙂

[marcphilipp]

I think we should go with a class-level annotation for now. We can always add a config parameter or ServiceLoader-based registration later. Let's call it @ResourceLocks and ResourceLockProvider, though, to stick to the terminology used in the Jupiter API.

[VladimirDmitrienko]

@marcphilipp got your point. I'm afraid @ResourceLocks could be a bit confusing since we already have such annotation. Don't you think?

[marcphilipp]

You're right, I forgot about the container annotation. I think @ResourceLockProvider would be good but then I don't know what we'd call the interface. Any ideas?

[VladimirDmitrienko]

What do you think about:

@ResourceLocksSource(MyResourceLocksProvider.class)

The same approach is used in @ArgumentsSource.

[marcphilipp]

I thought about that as well but concluded that it might be confusing since it's unrelated to the other source annotations.

[VladimirDmitrienko]

Fair enough.

There are two options coming in my mind.

1. What if we reuse existing @ResourceLocks annotation:

   public @interface ResourceLocks {
   
   Class<? extends ResourceLocksProvider> provider() default EmptyResourceLocksProvider.class;
       // or
   Class<? extends ResourceLocksProvider>[] providers() default {};
   }

2. @ResourceLocksFrom(MyResourceLocksProvider.class)

[marcphilipp]

What if we reuse existing @ResourceLocks annotation

That would break backward compatibility.

@ResourceLocksFrom(MyResourceLocksProvider.class)

Let's go with that. Renaming it as part of the PR should be easy enough if we come up with a better name.

[VladimirDmitrienko]

Agreed 👍 As always, naming is hard 😅

Just curious, what kind of compatibility will it break? I supposed adding a new method will be both a source and binary compatible change.

link

[marcphilipp]

Did you mean adding another attribute to the container annotation? If so, you may be right but I still think it would be confusing to appropriate it in this way.

[VladimirDmitrienko]

Got it, thank you 🙂 Will keep you updated on my work.

[VladimirDmitrienko]

Hi @marcphilipp

I've updated the PR and now it's ready for review. Looking forward to your feedback.

Btw, I've set API since to 5.12 but actually I'm not really sure in which release it will go, please, let me know and I'll update since values and add release notes.

[VladimirDmitrienko]

Hi @marcphilipp I would be happy to improve my PR and waiting for a feedback from your side. Maybe you have any at least approximate estimates of when could I expect that? Would be glad to receive any news on this one.

[marcphilipp]

Sorry for the delay! I was on vacation. I should be able to get to it this week. :crossed_fingers:

[VladimirDmitrienko]

@marcphilipp Thanks for the update 👍

[marcphilipp]

Reopening to track polishing work

[marcphilipp]

Thanks for your contribution, @VladimirDmitrienko! :+1:

[nickh-stripe]

little late to the party here, looking at the new experimental api for this: https://junit.org/junit5/docs/snapshot/api/org.junit.jupiter.api/org/junit/jupiter/api/parallel/ResourceLocksProvider.html

was it purposeful to exclude the test template / parameterised tests? seems you can only provide dynamically for class/method and nested class and not a templated or parameterised test which is what I was hoping for for my usecase?

In my scenario i want to use a property of the parameterised test input to define the resource lock key for a resource shared amongst a subset of tests..

[VladimirDmitrienko]

Hi @nickh-stripe

Actually ResourceLocksProvider can be used to define locks for parameterized tests. However, the same locks will be applied to each invocation of the test, regardless of the test input.

Let's consider such an example:

    @ParameterizedTest
    @ValueSource(strings = {"banana", "apple", "orange"})
    void fruitTest(String value) {
        // ...
    }

    static class Provider implements ResourceLocksProvider {

        @Override
        public Set<Lock> provideForMethod(Class<?> testClass, Method testMethod) {
            return Set.of(new Lock("fruitKey"));
        }
    }

The same lock applies to each invocation: fruitTest("banana"), fruitTest("apple"), fruitTest("orange").

It's impossible to define different locks based on the arguments, for example, setting the same lock for "banana" and "apple" but a different one for "orange". The reason is that locks are collected before test execution started: we first collect the locks and only then pass them to engine that executes tests.

---

Assigning locks depending on arguments could be achieved by splitting the test:

    @ParameterizedTest
    @ValueSource(strings = {"banana", "apple"})
    void bananaAndAppleTest(String value) {
        // ...
    }

    @ParameterizedTest
    @ValueSource(strings = {"orange"})
    void orangeTest(String value) {
        // ...
    }

    static class Provider implements ResourceLocksProvider {

        @Override
        public Set<Lock> provideForMethod(Class<?> testClass, Method testMethod) {
            if (testMethod.getName().equals("bananaAndAppleTest")) {
                return Set.of(new Lock("bananaAndAppleKey"));
            }
            else if (testMethod.getName().equals("orangeTest")) {
                return Set.of(new Lock("orangeKey"));
            }
            return Set.of();
        }
    }

[nickh-stripe]

thanks for the detailed explanation! the approach makes sense but is not possible for the shape of our problem unfortunately, splitting the tests based on their locks isn't really feasible unfortunately so i'll have to think about a different way to achieve it. 👍