OS Platform and Distribution (e.g., Linux Ubuntu 20.0): Mariner 5.15.111.1-1.cm2
JDK version: 17
Describe the problem
Background
Helix implemented a Customized-View feature for Venice specific requirements. In this feature, each instance within Helix Cluster updates their instance specific view and Helix controller aggregates view across all the cluster instances and presents at Cluster level. This information is used by Venice controllers to determine the health of their resources.
Problem Statement
When the instance/node in the cluster shuts down ungracefully, the instance has no way to update its own customized state. When the instance recovers, Helix aggregation logic reads the “stale” customized-state and updates the aggregated customized view, which results in the Venice router redirecting traffic to not-yet-recovered instance.
Instances maintain connection with ZK and create an Ephemeral node (LIVEInstance/) when it is up and running. When this ephemeral node disappears, Helix thinks that instance is down. Helix cannot distinguish - temporary disconnect, graceful shutdown and ungraceful shutdown.
Ideally in temporary disconnect - neither participant nor controller needs to do any changes to Customized-state or customized-view. In case of graceful shutdown, the participant instance will update its customized state. But in case of an ungraceful shutdown, we do not have any hooks today. This RFC is addressing this particular case.
Solution
Helix Participant can use the pre-existing callback provided by Helix-Manager. In Venice, SafeHelixManager.java - there is a method to addPrecallback(). This can be used by Venice to add the required logic to reset the customized-state. This callbacks are executed prior to creating the live instance ephemeral node.
In file: [HelixParticipationService.java]
helixManager.setLiveInstanceInfoProvider(liveInstanceInfoProvider); HelixStatusMessageChannel messageChannel =new
HelixStatusMessageChannel(helixManager, new HelixMessageChannelStats(metricsRepository, clusterName)); messageChannel.registerHandler(KillOfflinePushMessage.class, this);
...
asyncStart() <---- this is the one which does connection.
Interface to override:
@Override
public void onPreConnect() { ... }
The Helix will do the callback which the user has provided before creating the live-instance.
So in that code-path, Venice can add the logic to reset the customized-state for that instance.
Tracking information
No response
Code to reproduce bug
No response
What component(s) does this bug affect?
[ ] Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
[ ] Router: This is the stateless query-routing layer for serving read requests.
[X] Server: This is the component that persists all the store data.
[ ] VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.
[ ] VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
[ ] Thin Client: This is a stateless client users use to query Venice Router for reading store data.
[ ] Fast Client: This is a stateful client users use to query Venice Server for reading store data.
[ ] Da Vinci Client: This is an embedded, stateful client that materializes store data locally.
[ ] Alpini: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.
[ ] Samza: This is the library users use to make nearline updates to store data.
[ ] Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.
[ ] Scripts: These are the various ops scripts in the repo.
kvargha
commented
1 month ago
Willingness to contribute
No. I cannot contribute a bug fix at this time.
Venice version
0.4.139
System information
Describe the problem
Background
Helix implemented a Customized-View feature for Venice specific requirements. In this feature, each instance within Helix Cluster updates their instance specific view and Helix controller aggregates view across all the cluster instances and presents at Cluster level. This information is used by Venice controllers to determine the health of their resources.
Problem Statement
When the instance/node in the cluster shuts down ungracefully, the instance has no way to update its own customized state. When the instance recovers, Helix aggregation logic reads the “stale” customized-state and updates the aggregated customized view, which results in the Venice router redirecting traffic to not-yet-recovered instance. Instances maintain connection with ZK and create an Ephemeral node (LIVEInstance/) when it is up and running. When this ephemeral node disappears, Helix thinks that instance is down. Helix cannot distinguish - temporary disconnect, graceful shutdown and ungraceful shutdown. Ideally in temporary disconnect - neither participant nor controller needs to do any changes to Customized-state or customized-view. In case of graceful shutdown, the participant instance will update its customized state. But in case of an ungraceful shutdown, we do not have any hooks today. This RFC is addressing this particular case.
Solution
Helix Participant can use the pre-existing callback provided by Helix-Manager. In Venice, SafeHelixManager.java - there is a method to addPrecallback(). This can be used by Venice to add the required logic to reset the customized-state. This callbacks are executed prior to creating the live instance ephemeral node. In file: [HelixParticipationService.java]
Interface to override:
The Helix will do the callback which the user has provided before creating the live-instance. So in that code-path, Venice can add the logic to reset the customized-state for that instance.
Tracking information
No response
Code to reproduce bug
No response
What component(s) does this bug affect?
Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.Router: This is the stateless query-routing layer for serving read requests.Server: This is the component that persists all the store data.VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.Thin Client: This is a stateless client users use to query Venice Router for reading store data.Fast Client: This is a stateful client users use to query Venice Server for reading store data.Da Vinci Client: This is an embedded, stateful client that materializes store data locally.Alpini: This is the framework that fast-client and routers use to route requests to the storage nodes that have the data.Samza: This is the library users use to make nearline updates to store data.Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.Scripts: These are the various ops scripts in the repo.