search

wso2 / product-is

Welcome to the WSO2 Identity Server source code! For info on working with the WSO2 Identity Server repository and contributing code, click the link below.
http://wso2.github.io/
Apache License 2.0
746 stars 724 forks source link

Deadlock during concurrent SCIM user updates #21302

Open Yoshani opened 1 week ago

Yoshani commented 1 week ago

Describe the issue:

The following deadlock was observed due to concurrent SCIM user updates.

 ERROR {org.wso2.charon3.core.protocol.endpoints.AbstractResourceManager} - Error while updating attributes of user: D*********************************m org.wso2.charon3.core.exceptions.CharonException: Error while updating attributes of user: D*********************************m
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.resolveError(SCIMUserManager.java:405) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.updateUser(SCIMUserManager.java:1316) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.charon3.core.protocol.endpoints.UserResourceManager.updateWithPATCH(UserResourceManager.java:712) [org.wso2.charon3.core_4.0.18.jar:?]
at org.wso2.carbon.identity.scim2.provider.resources.UserResource.patchUser(UserResource.java:369) [classes/:?]
at jdk.internal.reflect.GeneratedMethodAccessor1182.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:201) [cxf-rt-frontend-jaxrs-3.5.5.jar:3.5.5]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:104) [cxf-rt-frontend-jaxrs-3.5.5.jar:3.5.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-core-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:304) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:281) [cxf-rt-transports-http-3.5.5.jar:3.5.5]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:661) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:425) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:357) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:294) [tomcat_9.0.82.wso2v1.jar:?]
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:147) [org.wso2.carbon.identity.context.rewrite.valve_1.9.5.jar:?]
at org.wso2.carbon.identity.context.rewrite.valve.OrganizationContextRewriteValve.invoke(OrganizationContextRewriteValve.java:123) [org.wso2.carbon.identity.context.rewrite.valve_1.9.5.jar:?]
at org.wso2.carbon.tomcat.ext.valves.SameSiteCookieValve.invoke(SameSiteCookieValve.java:38) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.identity.cors.valve.CORSValve.invoke(CORSValve.java:83) [org.wso2.carbon.identity.cors.valve_1.9.5.jar:?]
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:183) [org.wso2.carbon.identity.authz.valve_1.9.5.jar:?]
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:144) [org.wso2.carbon.identity.auth.valve_1.9.5.jar:?]
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:110) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:71) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:152) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.extension.identity.x509Certificate.valve.X509CertificateAuthenticationValve.invoke(X509CertificateAuthenticationValve.java:59) [org.wso2.carbon.extension.identity.x509Certificate.valve-1.1.10.jar:?]
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670) [tomcat_9.0.82.wso2v1.jar:?]
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:63) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.tomcat.ext.valves.RequestEncodingValve.invoke(RequestEncodingValve.java:49) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:137) [org.wso2.carbon.tomcat.ext_4.10.13.jar:?]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat_9.0.82.wso2v1.jar:?]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat_9.0.82.wso2v1.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]|Caused by: org.wso2.carbon.user.core.UserStoreException: Error occurred while updating string values to database.
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.updateProperties(UniqueIDJDBCUserStoreManager.java:3068) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.doSetUserAttributesWithID(UniqueIDJDBCUserStoreManager.java:1997) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.doSetUserClaimValuesWithID(AbstractUserStoreManager.java:903) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.doSetUserClaimValuesWithID(UniqueIDJDBCUserStoreManager.java:1971) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.setUserClaimValuesWithID(AbstractUserStoreManager.java:14207) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.setUserClaimValuesWithID(AbstractUserStoreManager.java:14142) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.updateUserClaims(SCIMUserManager.java:5391) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.updateUser(SCIMUserManager.java:1308) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
... 55 more|Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 256) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1695) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatementBatch(SQLServerPreparedStatement.java:2978) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtBatchExecCmd.doExecute(SQLServerPreparedStatement.java:2826) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7675) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4137) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:272) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:246) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2222) ~[mssql-jdbc-12.4.2.jre11.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.wso2.carbon.ndatasource.rdbms.CorrelationLogInterceptor$StatementProxy.invoke(CorrelationLogInterceptor.java:161) ~[org.wso2.carbon.ndatasource.rdbms_4.10.13.jar:?]
at com.sun.proxy.$Proxy55.executeBatch(Unknown Source) ~[?:?]
at jdk.internal.reflect.GeneratedMethodAccessor90.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118) ~[jdbc-pool_9.0.65.wso2v1.jar:?]
at com.sun.proxy.$Proxy55.executeBatch(Unknown Source) ~[?:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.updateProperties(UniqueIDJDBCUserStoreManager.java:3044) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.doSetUserAttributesWithID(UniqueIDJDBCUserStoreManager.java:1997) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.doSetUserClaimValuesWithID(AbstractUserStoreManager.java:903) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.doSetUserClaimValuesWithID(UniqueIDJDBCUserStoreManager.java:1971) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.setUserClaimValuesWithID(AbstractUserStoreManager.java:14207) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.setUserClaimValuesWithID(AbstractUserStoreManager.java:14142) ~[org.wso2.carbon.user.core-4.10.13.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.updateUserClaims(SCIMUserManager.java:5391) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
at org.wso2.carbon.identity.scim2.common.impl.SCIMUserManager.updateUser(SCIMUserManager.java:1308) ~[org.wso2.carbon.identity.scim2.common-3.4.80.0-ASGARDEOINTERNAL-hotfix-1.jar:?]
... 55 more|

Following is the query collected from the query hash presented in the deadlock xml file.


statement_text                                                                                                                                                                                                                                                   sql_handle                                                                                                                         statement_start_offset statement_end_offset query_hash        
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------- ---------------------- -------------------- ------------------
UPDATE UM_USER_ATTRIBUTE SET UM_ATTR_VALUE= @P0 WHERE UM_USER_ID=(SELECT UM_ID FROM UM_USER WHERE UM_USER_ID= @P1 AND UM_TENANT_ID= @P2 ) AND UM_ATTR_NAME= @P3 AND UM_PROFILE_ID= @P4 AND UM_TENANT_ID= @P5                                                    

**How to reproduce:**
Create a user in console
Concurrently update the user with SCIM
Yoshani commented 1 week ago

Fix: https://github.com/wso2/carbon-kernel/pull/4069