search

wso2 / product-is

Welcome to the WSO2 Identity Server source code! For info on working with the WSO2 Identity Server repository and contributing code, click the link below.
http://wso2.github.io/
Apache License 2.0
746 stars 724 forks source link

SelfRegisterTestCase.testSelfRegisterWithInvalidPassword integration test fails intermittently #21143

Open darshanasbg opened 3 weeks ago

darshanasbg commented 3 weeks ago

Describe the issue: Dependency updater build https://github.com/wso2/product-is/actions/runs/10997024103 failed with following result.

2024-09-23T16:48:58.6260198Z [ERROR] Tests run: 2598, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 5,492.891 s <<< FAILURE! - in TestSuite
2024-09-23T16:48:58.6262513Z [ERROR] testSelfRegisterWithInvalidPassword(org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase)  Time elapsed: 0.071 s  <<< FAILURE!
2024-09-23T16:48:58.6264826Z java.lang.AssertionError: Self register user password invalid expected:<400> but was:<409>
2024-09-23T16:48:58.6267385Z    at org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase.testSelfRegisterWithInvalidPassword(SelfRegisterTestCase.java:79)
2024-09-23T16:48:58.6269028Z 
2024-09-23T16:48:59.4934850Z [INFO] 
2024-09-23T16:48:59.4935329Z [INFO] Results:
2024-09-23T16:48:59.4935750Z [INFO] 
2024-09-23T16:48:59.4938151Z [ERROR] Failures: 
2024-09-23T16:48:59.4939567Z [ERROR]   SelfRegisterTestCase.testSelfRegisterWithInvalidPassword:79 Self register user password invalid expected:<400> but was:<409>
2024-09-23T16:48:59.4941053Z [INFO] 
2024-09-23T16:48:59.4941595Z [ERROR] Tests run: 2598, Failures: 1, Errors: 0, Skipped: 2

And the related logs for the test as follows:

2024-09-23T15:31:42.7509051Z INFO  [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - =================== Running the test method org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase.testSelfRegisterWithInvalidPassword ===================
2024-09-23T15:31:42.8215593Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.Assert.fail(Assert.java:89)
2024-09-23T15:31:42.8219288Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.Assert.failNotEquals(Assert.java:489)
2024-09-23T15:31:42.8226882Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.Assert.assertEquals(Assert.java:118)
2024-09-23T15:31:42.8254329Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.Assert.assertEquals(Assert.java:365)
2024-09-23T15:31:42.8257772Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase.testSelfRegisterWithInvalidPassword(SelfRegisterTestCase.java:79)
2024-09-23T15:31:42.8261635Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2024-09-23T15:31:42.8265246Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2024-09-23T15:31:42.8268718Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2024-09-23T15:31:42.8271645Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - java.base/java.lang.reflect.Method.invoke(Method.java:566)
2024-09-23T15:31:42.8274615Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
2024-09-23T15:31:42.8280634Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
2024-09-23T15:31:42.8283201Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.Invoker.invokeTestMethod(Invoker.java:842)
2024-09-23T15:31:42.8285692Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1166)
2024-09-23T15:31:42.8297598Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
2024-09-23T15:31:42.8300313Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
2024-09-23T15:31:42.8302741Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestRunner.runWorkers(TestRunner.java:1178)
2024-09-23T15:31:42.8305316Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestRunner.privateRun(TestRunner.java:757)
2024-09-23T15:31:42.8307547Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestRunner.run(TestRunner.java:608)
2024-09-23T15:31:42.8309758Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
2024-09-23T15:31:42.8312103Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
2024-09-23T15:31:42.8314604Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
2024-09-23T15:31:42.8316998Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunner.run(SuiteRunner.java:240)
2024-09-23T15:31:42.8319353Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
2024-09-23T15:31:42.8321797Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
2024-09-23T15:31:42.8324320Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestNG.runSuitesSequentially(TestNG.java:1158)
2024-09-23T15:31:42.8326631Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestNG.runSuitesLocally(TestNG.java:1083)
2024-09-23T15:31:42.8328776Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.testng.TestNG.run(TestNG.java:999)
2024-09-23T15:31:42.8331183Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:283)
2024-09-23T15:31:42.8334157Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
2024-09-23T15:31:42.8337024Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:120)
2024-09-23T15:31:42.8340163Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2024-09-23T15:31:42.8343410Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2024-09-23T15:31:42.8346268Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2024-09-23T15:31:42.8349070Z ERROR [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2024-09-23T15:31:42.8352709Z INFO  [org.wso2.carbon.automation.engine.testlisteners.TestManagerListener] - =================== On test failure

How to reproduce: This is a intermittent issue.. No clear steps to ATM to reproduce the issue.. Have to monitor, dig deep into possible root causes to find possible steps to reproduce the issue.

Expected behavior: Should be consistently pass the test..

Optional Fields

Related issues: Might related to another intermittent issue: https://github.com/wso2/product-is/issues/20261

indeewari commented 3 days ago

Failing due to the same reason

=================== On test failure org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase.testSelfRegisterWithInvalidPassword ===================

[ERROR] Tests run: 2633, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5,018 s <<< FAILURE! - in TestSuite
[ERROR] testSelfRegisterWithInvalidPassword(org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase)  Time elapsed: 0.124 s  <<< FAILURE!
java.lang.AssertionError: Self register user password invalid expected:<400> but was:<201>
    at org.wso2.identity.integration.test.rest.api.user.selfRegister.SelfRegisterTestCase.testSelfRegisterWithInvalidPassword(SelfRegisterTestCase.java:82)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   SelfRegisterTestCase.testSelfRegisterWithInvalidPassword:82 Self register user password invalid expected:<400> but was:<201>
[INFO] 
[ERROR] Tests run: 2633, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[ERROR] There are test failures.
madurangasiriwardena commented 3 days ago

Now the reason for new failure seems to be different from the previous reason. By the looks of it, it is similar to https://github.com/wso2/product-is/issues/20261 and password validation is somehow bypassed.

When the test is successful, it is supposed to log below log line (client errors are not supposed to be logged, which is a different problem), indicating the password validation has failed. But when the test case is failing, this validation is not properly working and passing unexpectedly.

ERROR {org.wso2.carbon.identity.input.validation.mgt.listener.InputValidationListener} - Failed to validate password for user. The minimum length of password should be 8.