wso2 / product-micro-integrator

The cloud-native configuration driven runtime that helps developers implement composite microservices.
https://wso2.com/integration/
Apache License 2.0
204 stars 221 forks source link

[Bug]: JMS inbound endpoints are failing once the cluster database issue occurred with update 11162 (U2 102) #3652

Closed veneerac closed 1 month ago

veneerac commented 1 month ago

Description

With update level 102. With this update an issue with JMS inbound endpoints. Whenever an issue occurs with the cluster database, JMS tasks fail with the following continuous error. Even after the database recovers, the same error is seen in the logs, and a server restart is required to resolve this issue.

Error ` ERROR {JMSTask} - JMS Polling Consumer is closed. Cannot execute the task

Steps to Reproduce

Steps to Replicate the Issue,

  1. Create the MI 3 Nodes cluster setup,
  2. Create the symlink for the carbon app folder for all of the 3 nodes, and deploy the scheduled task and JMS inbound endpoint
  3. Start the 3-node cluster setup. Then, we need to either forcefully suspend the Master node or add a delay to the database port of the RDBMS database that is running the tasks. We used the following command for that.

Add a delay rule for the RDBMS cluster database port

sudo tc qdisc add dev lo root handle 1: prio
sudo tc filter add dev lo parent 1: protocol ip prio 1 u32 match ip dport 3306 0xffff flowid 1:1
sudo tc qdisc add dev lo parent 1:1 handle 10: netem delay 15000ms

Remove delay's rule from the port sudo tc qdisc del dev lo root

Affected Version(s)

4.2.0.102

OS, DB, other environment details and versions

No response

Related issue(s) (optional)

https://github.com/wso2/micro-integrator/issues/3463

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response