Runtime-Modified @Test(description="some_new_description") value is not propagated at runtime

[Amro77]

HI,

Using TestNG7.0/Allure2.13.0,

I change @Test(description="....") during runtime to have a data-driven description passed to allure report, I found that old/unmodified test description is being passed to IInvokedMethodListener.beforeInvocation where:

public void beforeInvocation(IInvokedMethod method, ITestResult testResult)

calling method.getTestMethod().getDescription() returns the old unmodified test description

I had to alter Allure listener to get the exact description as follows:

ITestNGMethod testMethod = method.getTestMethod();
Test test = method.getTestMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
String test_description = getAnnotationValueMap(test).get("description").toString();
testMethod.setDescription(test_description);

where:

public static Map<String, Object> getAnnotationValueMap(Annotation annotation) {
        Object handler = Proxy.getInvocationHandler(annotation);
        Field f;
        try {
            f = handler.getClass().getDeclaredField("memberValues");
        } catch (NoSuchFieldException | SecurityException e) {
            throw new IllegalStateException(e);
        }
        f.setAccessible(true);

        Map<String, Object> memberValues;

        try {
            memberValues = (Map<String, Object>) f.get(handler);
        } catch (IllegalArgumentException | IllegalAccessException e) {
            throw new IllegalStateException(e);
        }
        return memberValues;
}

I think this should be done in TestNG.

and here is how I change test description

   public static void updateTestDescription(Method method, String newDescription) {
        Test test = method.getAnnotation(Test.class);
        changeAnnotationValue(test, "description", newDescription);
    }

    @SuppressWarnings("unchecked")
    public static Object changeAnnotationValue(Annotation annotation, String key, Object newValue){
        Object handler = Proxy.getInvocationHandler(annotation);
        Field f;
        try {
            f = handler.getClass().getDeclaredField("memberValues");
        } catch (NoSuchFieldException | SecurityException e) {
            throw new IllegalStateException(e);
        }
        f.setAccessible(true);
        Map<String, Object> memberValues;
        try {
            memberValues = (Map<String, Object>) f.get(handler);
        } catch (IllegalArgumentException | IllegalAccessException e) {
            throw new IllegalStateException(e);
        }
        Object oldValue = memberValues.get(key);
        if (oldValue == null || oldValue.getClass() != newValue.getClass()) {
            throw new IllegalArgumentException();
        }
        memberValues.put(key,newValue);

        return oldValue;
    }

[krmahadevan]

@Amro77 - TestNG does not alter any annotation objects using reflection. It basically creates something like a proxy object and then sets and retrieves values to to/from the proxy objects. That is why even though you changed the annotation values using reflection, you don't see them being reflected in TestNG.

There are a couple of things that needs to be done here to actually be able to alter the description on the fly in allure reports.

1. You would need to create a package named io.qameta.allure.testng
2. Duplicate the class AllureTestNg
3. Now alter the public void onTestStart(final ITestResult testResult) and add the below two lines to the start of this method (The lines that are residing within the comment Custom code

  @Override
  public void onTestStart(final ITestResult testResult) {
    // Custom code.
    ITestNGMethod testngMethod = testResult.getMethod();
    if (testngMethod.isDataDriven()) {
      String desc = "With_Data_" + Arrays.toString(testResult.getParameters());
      testngMethod.setDescription(desc);
    }
    //Custom code.

    Current current = currentTestResult.get();
    if (current.isStarted()) {
      current = refreshContext();
    }
    current.test();
    final String uuid = current.getUuid();
    final String parentUuid = getUniqueUuid(testResult.getTestContext());

    startTestCase(testResult, parentUuid, uuid);

    Optional.of(testResult)
        .map(ITestResult::getMethod)
        .map(ITestNGMethod::getTestClass)
        .ifPresent(clazz -> addClassContainerChild(clazz, uuid));
  }

This should do it and you should now start seeing the description which is based on what you are altering it to (In my case, I was altering it to honor the method parameters )

Here's a screen shot of that

Once TestNG 7.2.0 gets released, you would be able to disable SPI powered listeners. Refer here to learn how to do that. So you could disable the mandatory listener io.qameta.allure.testng.AllureTestNg, extend it, override just the onTestStart() method and add the above mentioned code.

[Amro77]

Thanks a lot @krmahadevan !!

I had already found a workaround without duplicating AllureTestNg, from Allure side by extending AllureLifecycle and overriding startTestCase, but I see that Allure was doing the right action, it was just calling ITestNGMethod.getDescription() to get description (Allure had another problem also that would result in new description to overwrite the previous one of the same method, I solved it too Alhamdulillah), meaning that if TestNG was passing the runtime description of the method then there would be no issue, in other words TestNG was passing outdated description in ITestNGMethod object to the listener.

[krmahadevan]

@Amro77 - I dont think TestNG is passing any outdated information because TestNG parses annotations, creates proxy objects well early in the cycle and after that it just works with the proxy object. So after TestNG parses the annotations, it no longer keeps going back to check if the annotations were altered via reflection.

So in your case, if you are altering the description (which if you see in my example, I am also altering it via proper TestNG provided APIs), then it should be done such that, it happens before Allure reads the description, which is what was going wrong in your case.

[Amro77]

@krmahadevan

So after TestNG parses the annotations, it no longer keeps going back to check if the annotations were altered via reflection.

So how am I supposed to modify @Test(description) description at run time in test class? Reporting framework should not alter description, I believe its responsibility should be reading it without a change.

[krmahadevan]

@Amro77

So how am I supposed to modify @test(description) description at run time in test class?

You cannot do that, because Allure Reports retrieves the description in onStart() method itself, which gets invoked even before the test method gets invoked.