search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Clusters stopped getting workers #152

Closed mrocklin closed 3 years ago

mrocklin commented 3 years ago

I've been bringing up clusters all day today (thanks by the way, this is great). However recently my clusters have stopped getting workers. My worker count sits at 0/50 for 5-10 minutes.

Scheduler name: mrocklin-cbf03a69-a

I've just switched from hosted AWS to hosted GCP to see if that resolves the problem.

mrocklin commented 3 years ago

Workers are coming up in GCP, although slowly.

FabioRosado commented 3 years ago

I've had a quick look on the logs and it seems that the reason why workers stopping showing is because AWS didn't have any more instances available.

We have an issue to handle the InsufficientInstanceCapacity better in the short term and then rework our VPC setup should solve most of this problem

mrocklin commented 3 years ago

Good to know. It was great being able to switch clouds in this case.