search

camunda / camunda-bpm-platform

Flexible framework for workflow and decision automation with BPMN and DMN. Integration with Quarkus, Spring, Spring Boot, CDI.
https://camunda.com/
Apache License 2.0
4.11k stars 1.55k forks source link

Camunda - Lock on BPM file - #3562

Closed kunalgithub closed 1 year ago

kunalgithub commented 1 year ago

Environment (Required on creation)

7.15 - embedded camunda spring boot 2.3 RELEASE

Description (Required on creation; please attach any relevant screenshots, stacktraces, log files, etc. to the ticket)

"camundaTaskExecutor-26" #335 prio=5 os_prio=0 tid=0x00007f041000d000 nid=0x300c waiting for monitor entry [0x00007f0404bf6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82)
        - waiting to lock <0x00000006438a0010> (a java.lang.Class for spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector)
        at spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:125)
        at spinjar.com.sun.xml.bind.v2.runtime.reflect.Accessor$GetterSetterReflection.optimize(Accessor.java:399)
        at spinjar.com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:94)
        at sun.reflect.GeneratedConstructorAccessor226.newInstance(Unknown Source)

Steps to reproduce (Required on creation)

  1. Define an AsyncMethod
  2. Inside Async - Initiate Runtime.startProcessInstaceByKey()

Observed Behavior (Required on creation)

As stated in stack trace

Expected behavior (Required on creation)

Root Cause (Required on prioritization)

Solution Ideas

Hints

Links

Breakdown

### Pull Requests

Dev2QA handover

kunalgithub commented 1 year ago 
Full thread dump OpenJDK 64-Bit Server VM (25.372-b07 mixed mode):
"Keep-Alive-Timer" #345 daemon prio=8 os_prio=0 tid=0x00007f04d8026800 nid=0x3068 waiting on condition [0x00007f04045f1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
    at java.lang.Thread.run(Thread.java:750)
   Locked ownable synchronizers:
    - None
"camundaTaskExecutor-28" #337 prio=5 os_prio=0 tid=0x00007f0410011800 nid=0x300e waiting for monitor entry [0x00007f04041ea000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82)
    - waiting to lock <0x00000006438a0010> (a java.lang.Class for spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector)
    at spinjar.com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:125)
    at spinjar.com.sun.xml.bind.v2.runtime.reflect.Accessor$GetterSetterReflection.optimize(Accessor.java:399)
    at spinjar.com.sun.xml.bind.v2.runtime.property.SingleMapNodeProperty.<init>(SingleMapNodeProperty.java:106)
    at sun.reflect.GeneratedConstructorAccessor317.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at spinjar.com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128)
    at spinjar.com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:181)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:509)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:528)
    at spinjar.com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:105)
    at sun.reflect.GeneratedConstructorAccessor226.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at spinjar.com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128)
    at spinjar.com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:181)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:509)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:528)
    at spinjar.com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:105)
    at sun.reflect.GeneratedConstructorAccessor226.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at spinjar.com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128)
    at spinjar.com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:181)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:509)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:326)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:141)
    at spinjar.com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1157)
    at spinjar.com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145)
    at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at spinjar.javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:263)
    at spinjar.javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:250)
    at spinjar.javax.xml.bind.ContextFinder.find(ContextFinder.java:447)
    at spinjar.javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:652)
    at spinjar.javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:599)
    at org.camunda.spin.impl.xml.dom.format.spi.DefaultJaxBContextProvider.getContext(DefaultJaxBContextProvider.java:39)
    at org.camunda.spin.impl.xml.dom.format.spi.DefaultJaxBContextProvider.createUnmarshaller(DefaultJaxBContextProvider.java:58)
    at org.camunda.spin.impl.xml.dom.format.DomXmlDataFormatMapper.getUnmarshaller(DomXmlDataFormatMapper.java:171)
    at org.camunda.spin.impl.xml.dom.format.DomXmlDataFormatMapper.mapInternalToJava(DomXmlDataFormatMapper.java:119)
    at org.camunda.spin.impl.xml.dom.format.DomXmlDataFormatMapper.mapInternalToJava(DomXmlDataFormatMapper.java:157)
    at org.camunda.spin.plugin.impl.SpinObjectValueSerializer.deserializeFromByteArray(SpinObjectValueSerializer.java:101)
    at org.camunda.bpm.engine.impl.variable.serializer.AbstractObjectValueSerializer.deserializeFromByteArray(AbstractObjectValueSerializer.java:119)
    at org.camunda.bpm.engine.impl.variable.serializer.AbstractSerializableValueSerializer.readValue(AbstractSerializableValueSerializer.java:83)
    at org.camunda.bpm.engine.impl.variable.serializer.AbstractSerializableValueSerializer.readValue(AbstractSerializableValueSerializer.java:31)
    at org.camunda.bpm.engine.impl.persistence.entity.util.TypedValueField.getTypedValue(TypedValueField.java:115)
    at org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.getTypedValue(VariableInstanceEntity.java:282)
    at org.camunda.bpm.engine.impl.core.variable.scope.AbstractVariableScope.getValueFromVariableInstance(AbstractVariableScope.java:142)
    at org.camunda.bpm.engine.impl.core.variable.scope.AbstractVariableScope.getVariable(AbstractVariableScope.java:129)
    at org.camunda.bpm.engine.impl.core.variable.scope.AbstractVariableScope.getVariable(AbstractVariableScope.java:125)
    at com.*****.lender.eligibility.workflow.PrepareFinalOfferParametersOutput.execute(PrepareFinalOfferParametersOutput.java:38)
    at org.camunda.bpm.engine.impl.bpmn.delegate.JavaDelegateInvocation.invoke(JavaDelegateInvocation.java:40)
    at org.camunda.bpm.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:58)
    at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocationInContext(DefaultDelegateInterceptor.java:92)
    at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:63)
    at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior$3.call(ServiceTaskDelegateExpressionActivityBehavior.java:118)
    at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior$3.call(ServiceTaskDelegateExpressionActivityBehavior.java:102)
    at org.camunda.bpm.engine.impl.bpmn.behavior.AbstractBpmnActivityBehavior.executeWithErrorPropagation(AbstractBpmnActivityBehavior.java:90)
    at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior.performExecution(ServiceTaskDelegateExpressionActivityBehavior.java:127)
    at org.camunda.bpm.engine.impl.bpmn.behavior.TaskActivityBehavior.execute(TaskActivityBehavior.java:69)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:61)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:50)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueIfExecutionDoesNotAffectNextOperation(PvmExecutionImpl.java:2039)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:42)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:31)
    at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:131)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:111)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:86)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:634)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:608)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$6.callback(PvmExecutionImpl.java:1978)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$6.callback(PvmExecutionImpl.java:1975)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueExecutionIfNotCanceled(PvmExecutionImpl.java:2045)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1994)
    at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1975)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:60)
    at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:30)
    at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:66)
    at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:131)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:111)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:86)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:76)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:643)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:618)
    at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:62)
    at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:131)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:111)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:86)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:76)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:643)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:618)
    at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:62)
    at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:99)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:131)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:111)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:86)
    at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:76)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:643)
    at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:618)
yanavasileva commented 1 year ago

Hello @kunalgithub,

Thank you for reaching out to us with this and your interest in the product.

As first step, I would like to point out:

Could you please try to reproduce the issue with currently supported environment setup. For example, you can try using Camunda Platform 7.19 with Spring Boot version 2.7 and provide a minimal project example that reproduces the issue in order to speed the investigation of the issue.

Also could you please provide more details on:

  1. Did you face the issue when you first implemented the process application or it was after update? If the latter what kind of update or modification it was?
  2. How often the issue occurs?
  3. Does it occur only for a specific process model or for multiple?

Further you can read about Configuring Spin Data Formats in our User guide. ^3

Best regards, Yana

kunalgithub commented 1 year ago

this issue was occurring as a side effect of creating too many workflows per second- basically concurrent same workflow

yanavasileva commented 1 year ago

Thank you for getting back to us with your finding. I will close the ticket now. You can reach out with usage questions in our forum.