From Gerald: "I had to rerun terraform to reprovision SPS to resolve this issue. so it seems that the load balancer that frontends sps-api can be unresponsive if the nodes that support it are shut down. in my case, sending the prewarm SPS API num_nodes=1 terminated 2 of the verdi compute nodes which then destabilized the ELB. I think updating the terraform so that the SPS API's ELB utilizes nodes from the SPS API node group or the default node group would mitigate this issue."
From Gerald: "I had to rerun terraform to reprovision SPS to resolve this issue. so it seems that the load balancer that frontends sps-api can be unresponsive if the nodes that support it are shut down. in my case, sending the prewarm SPS API num_nodes=1 terminated 2 of the verdi compute nodes which then destabilized the ELB. I think updating the terraform so that the SPS API's ELB utilizes nodes from the SPS API node group or the default node group would mitigate this issue."