Closed jaronoff97 closed 1 month ago
Some agents that are not directly informed of health changes should also use this for component health reporting.
@jaronoff97 Can you clarify this? I am not sure I fully understand what this means.
@tigrannajaryan right now the bridge is not informed of the health of the agents it monitors, but instead uses the heartbeat interval to query the component health in the cluster. I figured that other agents that are doing similar interval-based health operations should do the same. I can clarify more in the spec to make that clear.
@andykellr if you would like to take a look, otherwise I will merge it.
You can merge. I've been following along and looks great.
great work, thank you @jaronoff97!
This PR introduces a client side heartbeat that can be set via the OpAMP Connection Settings. After discussing more in the past SIG meeting, I worked through the following pro/con for a client side vs server side heartbeat.
closes https://github.com/open-telemetry/opamp-spec/issues/183
Client Side
A client side heartbeat would be negotiated between the client and server, where the client's default heartbeat is set to 30s. On connect, the server has an option to entirely disable heartbeats by setting an explicit 0 for the field. The server can also offer a different heartbeat interval depending on its infrastructure's needs. After connection, the agent will begin a timer and each N seconds will send a message that minimally contains its instance id. An HTTP connection will use this interval for its polling as well. Some agents that are not directly informed of health changes should also use this for component health reporting.
Pros
Cons
Server Side
A server side heartbeat would simply be a part of opamp-go and would require no changes to the spec to allow this to work. The server every N seconds would send an empty message to the client to keep the socket connection active.
Pros
Cons
I think given the pros and cons of the above, I prefer an Agent heartbeat over a server heartbeat. If we need a new proto change anyway to introduce a heartbeat flag, I think the client approach is more effective. Furthermore, this change helps provide guidance for agents that are not informed of status updates. By setting an explicit heartbeat for the client, the server can increase the granularity of the agent's status updates. The client heartbeat approach also matches the design for a conventional deadman's switch – something that is constantly sending a signal out for a receiver to detect only when that signal is no longer received. Flipping that design removes that guarantee and weakens the overall feature.
Finally, the server would also be able to explicitly disconnect misbehaving clients and force them to reconnect with new settings. If the server were to not receive a heartbeat within its set window, the server could initiate a disconnect to gracefully close the client. This approach would not work as well for a server heartbeat as it would need to cancel the initial server to agent message.
Open Questions
References