I added a task to a chronos, this task was faulty ( typo in docker image tag ), and chronos ( latest 2.x version ) tried to launch it many times silently (no loop protection, no errors in sentry).
After a while, chronos became unstable, and after killing it, it can no longer be started.
Each time a task launch, a new entry is added in /chronos/state/state. With a faulty job, it loops until it raise zookeeper node size limits and chronos starts to become unstable.
Marathon was facing a similar issue and changed it's zookeeper storage layout since version 1.3.
Chronos should prevent faulty job to loop, report launch errors in sentry / slack /... , and adopt a nested storage layout.
For reference, to fix a bloated state:
Increase jute.maxbuffer to workaround zk limits and retrieve the huge children list
JVMFLAGS="-Djute.maxbuffer=33554430" zkCli -server 127.0.0.1:2181 ls /chronos/state/state > chronos-state.txt
Use sed / tr / grep to filter and reformat entries related to the faulty tasks, you must have one delete /chronos/state/state/$taskname per line
Pipe the result back to zkCli. Be patient, it takes a while to complete.
I added a task to a chronos, this task was faulty ( typo in docker image tag ), and chronos ( latest 2.x version ) tried to launch it many times silently (no loop protection, no errors in sentry).
After a while, chronos became unstable, and after killing it, it can no longer be started.
Each time a task launch, a new entry is added in
/chronos/state/state
. With a faulty job, it loops until it raise zookeeper node size limits and chronos starts to become unstable.Marathon was facing a similar issue and changed it's zookeeper storage layout since version 1.3.
Chronos should prevent faulty job to loop, report launch errors in sentry / slack /... , and adopt a nested storage layout.
For reference, to fix a bloated state:
Increase
jute.maxbuffer
to workaround zk limits and retrieve the huge children listJVMFLAGS="-Djute.maxbuffer=33554430" zkCli -server 127.0.0.1:2181 ls /chronos/state/state > chronos-state.txt
Use sed / tr / grep to filter and reformat entries related to the faulty tasks, you must have one
delete /chronos/state/state/$taskname
per linePipe the result back to zkCli. Be patient, it takes a while to complete.