We are using auto start/stop PS script which powers on/off the AWS EC2 instances then they are/aren't needed. Periodically some nodes "stuck" with "Node State" - Starting or Draining and "Node Health" - Transitional. These nodes could not be started or stopped by script anymore and they need to be removed and readded to the cluster.
For example at the moment one of the node which is highlighted in printscreens, was powered on, but can't be powered off by script as it's shows Starting Transitional. As there is now active jobs at the moment, this node wasting our money.
Steps to Reproduce
Power on and off the Compute node in the cluster.
Expected Results
Node State - "Online" and Node Health - "Ok" then node is running. Node State - "Offline" and Node Health - "Error" then node is powered off.
Problem Description
We are using auto start/stop PS script which powers on/off the AWS EC2 instances then they are/aren't needed. Periodically some nodes "stuck" with "Node State" - Starting or Draining and "Node Health" - Transitional. These nodes could not be started or stopped by script anymore and they need to be removed and readded to the cluster. For example at the moment one of the node which is highlighted in printscreens, was powered on, but can't be powered off by script as it's shows Starting Transitional. As there is now active jobs at the moment, this node wasting our money.
Steps to Reproduce
Power on and off the Compute node in the cluster.
Expected Results
Node State - "Online" and Node Health - "Ok" then node is running. Node State - "Offline" and Node Health - "Error" then node is powered off.
Actual Results