Currently the spark operator check for state updates every 3 minutes. This is a UX problem for tooling like Open Data Hub because if a user hits the JH spawner, it can take up to 3 minutes for a spark cluster to actually spin up. So a user will get their notebook open, and try connecting to a spark cluster, and be confused when the connection to spark fails.
Ideally, it would be good to make this interval configurable. Or if for some reason configurability is a problem, then make the hard-coded interval shorter.
erikerlandson
commented
4 years ago
Currently the spark operator check for state updates every 3 minutes. This is a UX problem for tooling like Open Data Hub because if a user hits the JH spawner, it can take up to 3 minutes for a spark cluster to actually spin up. So a user will get their notebook open, and try connecting to a spark cluster, and be confused when the connection to spark fails.
Ideally, it would be good to make this interval configurable. Or if for some reason configurability is a problem, then make the hard-coded interval shorter.