dhananjay12
commented
1 year ago The problem is with any kind of exception we get in the producer, after which it makes the readiness probe DOWN. Another way to produce the issue is
@GetMapping("/test")
public void test() {
JSONObject jsonObject = new JSONObject();
for (int i = 0; i <= 3000; i++) {
String key = "description" + i;
String value = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque cursus, nisl sed vestibulum tempus, " +
"lectus leo venenatis massa, ut ullamcorper libero enim sit amet mi. Fusce aliquam aliquet augue, " +
"vitae pellentesque ex dignissim quis. Sed et neque ligula. Vivamus elementum varius lobortis. Sed " +
"efficitur, lorem vel efficitur faucibus, libero ex iaculis nisi, id consectetur massa orci ac tortor. " +
"Nullam malesuada lectus enim, vitae bibendum ex pulvinar nec. Nulla facilisi. Suspendisse potenti.";
jsonObject.put(key, value);
}
streamBridge.send("hello-out-0", jsonObject);
}This is a huge object and obviously will throw an exception.
The readiness probe is down and will only be up after a successful request. We are in a similar situation like before where in K8s, the traffic is stopped to the POD when readiness probes returns DOWN.
In the Code, I see the instrumentation is added to every producer Seeing the Tests, while its marked as DOWN, I was wondering if a successful subsequent request is the only way to bring it UP?
Checking with other binders like Kafka, Rabbit, they typically check only connection, and recovers automatically, once the connection is back up.
I guess our expectation would be:
- We do want it to go down on failed connection, but it should try to reestablish connection on subsequent readiness probe calls.
- For other kind of exceptions in producers like large message needs to be skipped from readiness health
Netyyyy
awsule-cvs
saragluna
Background
We are using service bus binder which pushes message to Azure service bus topics. The app is running in a K8s environment.
spring-cloud-azure-stream-binder-servicebus - 4.4.1 Spring boot -2.7.7
There are unknown situation where the connectivity for the app and azure-service-bus is lost, resulting in readiness probe to be down.
In the logs I can see that the expected error
Spring actuator configurations for readiness/liveness:
readiness.show-details=alwaysis only to debug the application.Because the readiness probe is DOWN, K8s stops sending traffic to the POD.
Issue
After the connection is restored, the readiness probe remains down and does not recover. In this state, even though a successful request is technically possible, it never occurs because Kubernetes has halted traffic to the affected container.
When reproducing this issue locally, if I send a request to the service, it establishes a connection and sends the message to the service bus, causing the Health Endpoint status to return to UP. However, the situation differs in Kubernetes, where the traffic is blocked to the container, leaving it in a persistent state of unavailability.
I did find this class in the azure-binder class
Is there any other configuration you would suggest to tackle this scenario?
Setup
The only way I could reproduce this situation locally is by running a simple app when the connection with the service bus is stable. I make a few successful requests and then intentionally disable the internet connection on my laptop. When attempting to make another request, it throws a retry exception, causing the readiness probe to fail (and the application to become unavailable if it was in K8s env). Even after restoring the internet connection, the app's readiness probe never recovers.
Simple app - https://github.com/dhananjay12/learn-azure/tree/main/scs-service-bus NOTE - For connection we are using
service principalway to connect to the service-bus topic.