search

apache / helix

Mirror of Apache Helix
Apache License 2.0
464 stars 225 forks source link

Pinot Controller leader re-election not triggered #2915

Open cypherean opened 3 weeks ago

cypherean commented 3 weeks ago

Describe the bug

We faced an issue in Pinot production setup wherein the controller leader node went down but re-election was not triggered. This lead to segment upload errors for tables trying to commit a segment, instances being marked as unavailable for the segment and finally queries failing for the tables with segments unavailable error.

Timeline for this was as follows:

  1. A GET call for a large table's segments' metadata tables/<tablename>/segments/<segmentName>/metadata?columns=* spawns ~75k threads. This cause a huge memory spike and heap to go out of memory (heap size being 128GB here), maybe crashing the node. We suspect it was because of reload status button which triggers the segment metadata call Screenshot 2024-09-12 at 7 11 38 PM

  2. The node's 2 zk sessions time out at 17:19:56, the node tries to reestablish connection but it keeps emitting metrics as a leader until 17:29:x Screenshot 2024-09-12 at 7 11 05 PM

  3. The health check for node starts failing around 17:20:x, but standby controller nodes keep polling and getting failed leader's session ID as leader until 18:10 when we triggered a force replacement of error node Screenshot 2024-09-12 at 7 19 24 PM

Instance 123 is not leader of cluster production-cluster due to current session 702147796290178 does not match leader session 702147796290171

Expected behavior

Ideally the re-election should've triggered around 17:20.

Additional context

Helix version - 1.0.4 Pinot version - 1.0.x Pinot issue - https://github.com/apache/pinot/issues/13990

junkaixue commented 17 hours ago

I remember it was fixed by Jiajun before. The root cause was Helix reading the session data from the ZNode stored content instead of stats of the LEADER ZNode itself.

For version 1.3+, it has been fixed.