search

wso2 / product-micro-integrator

The cloud-native configuration driven runtime that helps developers implement composite microservices.
https://wso2.com/integration/
Apache License 2.0
204 stars 221 forks source link

[Bug]: OpenTelemetry- java.util.ConcurrentModificationException #3656

Closed DaniduWijendra closed 1 month ago

DaniduWijendra commented 1 month ago

Description

When using JMS proxy to produce and consume messages, observed following error intermittently, when enabled opentracing.

ERROR {org.apache.synapse.mediators.base.SequenceMediator} - {proxy:JMSConsumerProxy} Runtime error occurred while mediating the message java.util.ConcurrentModificationException
    at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1511)
    at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1544)
    at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1542)
    at java.base/java.util.AbstractMap.toString(AbstractMap.java:551)
    at org.apache.synapse.aspects.flow.statistics.tracing.opentelemetry.management.helpers.SpanTagger.setSpanTags(SpanTagger.java:105)
    at org.apache.synapse.aspects.flow.statistics.tracing.opentelemetry.stores.SpanStore.finishSpan(SpanStore.java:124)
    at org.apache.synapse.aspects.flow.statistics.tracing.opentelemetry.management.handling.span.SpanHandler.finishSpan(SpanHandler.java:358)
    at org.apache.synapse.aspects.flow.statistics.tracing.opentelemetry.management.handling.span.SpanHandler.handleCloseEvent(SpanHandler.java:315)
    at org.apache.synapse.aspects.flow.statistics.tracing.opentelemetry.management.handling.span.SpanHandler.handleCloseEntryEvent(SpanHandler.java:303)
    at org.apache.synapse.aspects.flow.statistics.collectors.CloseEventCollector.closeEntryEvent(CloseEventCollector.java:100)
    at org.apache.synapse.aspects.flow.statistics.collectors.CloseEventCollector.closeEntryEvent(CloseEventCollector.java:53)
    at org.apache.synapse.mediators.AbstractMediator.reportCloseStatistics(AbstractMediator.java:497)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:111)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:74)
    at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
    at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:240)
    at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
    at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
    at org.apache.axis2.transport.jms.JMSMessageReceiver.processThoughEngine(JMSMessageReceiver.java:225)
    at org.apache.axis2.transport.jms.JMSMessageReceiver.onMessage(JMSMessageReceiver.java:129)
    at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.handleMessage(ServiceTaskManager.java:758)
    at org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:561)
    at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Steps to Reproduce

[mediation]
inbound.core_threads = 200
inbound.max_threads = 1000
synapse.enable_xpath_dom_failover="true"
synapse.disable_custom_replace_regex = "@@@@"
flow.statistics.enable=true
flow.statistics.capture_all=true
synapse.global_timeout_interval = "180000"
stat.tracer.collect_payloads= true
stat.tracer.collect_mediation_properties= true

[synapse_properties]
'synapse.commons.enableXmlNullForEmptyElement' = "false"
'synapse.json.to.xml.processing.instruction.enabled' = true

[opentelemetry]
enable = true
logs = true
type = "jaeger"
host = "localhost"
port = 14250

[transport.jms]
sender_enable = true

[[transport.jms.listener]]
name = "myQueueConnectionFactory"
parameter.initial_naming_factory = "org.wso2.andes.jndi.PropertiesFileInitialContextFactory"
parameter.provider_url = "conf/jndi.properties"
parameter.connection_factory_name = "QueueConnectionFactory"
parameter.connection_factory_type = "queue"

[transport.jndi.connection_factories]
'connectionfactory.QueueConnectionFactory' = "amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5675'"
'connectionfactory.TopicConnectionFactory' = "amqp://admin:admin@clientID/carbon?brokerlist='tcp://localhost:5675'"

[1] https://bajpushpakumara.medium.com/publish-subscribe-with-jms-using-wso2-enterprise-integrator-14501ba335a

Affected Version(s)

MI-4.2.0

OS, DB, other environment details and versions

No response

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response