exacaster / lighter

REST API for Apache Spark on K8S or YARN
MIT License
91 stars 21 forks source link

Lighter sessions always display session states as IDLE state even during active query execution #1061

Open HarshithaRmAkamai opened 4 months ago

HarshithaRmAkamai commented 4 months ago

Lighter sessions show an IDLE state even while queries are running, though the expectation was to see BUSY during execution and IDLE when not executing.

Additionally, in lighter batches, I see the following states: NOT_STARTED, STARTING, IDLE, BUSY, SHUTTING_DOWN, ERROR, DEAD, KILLED, and SUCCESS. I only see a few of these states being applied to sessions, such as NOT_STARTED (when the session is about to start), STARTING (when the session has started), IDLE (where it remains most of the time, both during execution and when not executing), and ERROR. I would like to know if all these states are supposed to apply to sessions as well.

Screenshot 2024-06-18 at 2 57 07 PM