Closed PehrGit closed 7 months ago
If you let the app run after throwing, it will report unhealthy right away. Because you hold the execution, it drops on timeout (20 sec) and then the function to report unhealthy sub is called. When you hold there, the exception is not bubbled up to the sub, and it doesn't crash the event handling callback, so it just waits. Then, it times out and drops.
I'd expect you to see log messages from this:
Log.WarnLog?.Log(exception, "Dropped: {Reason}", reason);
It's the second line of the Dropped
function, which is called on subscription drop.
I was also writing wrong about resubscribe. Until it actually resubscribes, the subscription should remain dropped.
@PehrGit shall we conclude this as resolved since I merged your PR?
Yes let's do that. My initial assumption, that an exception thrown during handling should make the check unhealthy, was incorrect. As you pointed out in the Discord, the health check is intended to ensure connectivity to the database. To verify that events are being handled, we should use the metrics (see #309).
Thanks for assisting and clarifying that!
Describe the bug I am trying to get the health checks working, as described here: https://eventuous.dev/docs/subscriptions/subs-diagnostics/#health-checks.
I have added the required code but when an exception occurs, the health status remains "healthy".
To Reproduce I have modified the dotnet-sample repo to always throw an exception on the RoomBooked event. Steps to reproduce the behavior:
Expected behavior I expect the state to become "Unhealthy" because the subscription has been dropped. I have set breakpoints in the SubscriptionHealthCheck class and I see that ReportHealthy is called at the time of Subscribe, but the ReportUnhealthy is never called.
Screenshots![image](https://github.com/Eventuous/eventuous/assets/3782086/3bd70e05-af44-4805-9582-173d3746a75a)
Desktop (please complete the following information):
Additional context
Log output:
To make matters even stranger, I noticed that the breakpoint in
SubscriptionHealthCheck.ReportUnhealthy()
, which I am expecting to be hit, will be hit when I leave the debugger hanging on the exception for 20 seconds or more, but not before that. Videos showing the behaviour:https://github.com/Eventuous/eventuous/assets/3782086/5f3fadab-bef6-482d-ab3e-9b291b406839
https://github.com/Eventuous/eventuous/assets/3782086/16772e99-61cf-4a10-882e-238b6f91c44f