spotty-cloud / spotty

Training deep learning models on AWS and GCP instances
https://spotty.cloud
MIT License
493 stars 43 forks source link

Incomplete formatting of an error message #51

Closed vadimkantorov closed 5 years ago

vadimkantorov commented 5 years ago

Got this:

Error:
------
Spot price for the "%s" availability zone not found.

The format argument was omitted here: https://github.com/apls777/spotty/blob/master/spotty/providers/aws/helpers/spot_prices.py#L27

vadimkantorov commented 5 years ago

This happens when the automatically suggested AZ zone does not have spot instances. Debug print:

{'us-east-1a': 3.672, 'us-east-1f': 3.672, 'us-east-1c': 3.672, 'us-east-1b': 3.672}
Error:
------
Spot price for the "us-east-1d" availability zone not found.
apls777 commented 5 years ago

Thanks, I'll fix formatting, but I'm not sure I understand how exactly you're getting this error. In the code I see only 2 paths:

  1. You already have EBS volumes created in the us-east-1d zone and you're using the maxPrice parameter, then you see this error before the instance is actually started.
  2. You already have EBS volumes created in the us-east-1d zone and a Spot Instance was successfully started in this zone, but AWS API didn't return a price for this zone for some reason. That would be very weird.

Is it the first case or do I miss something?

vadimkantorov commented 5 years ago

It was case (1). I didn't explicitly specify the AZ when I was creating the EBS volume and the temporary On-Demand instance. Then I discovered that the AZ selected by default didn't have p3.x8large spot instances.

vadimkantorov commented 5 years ago

I meant to say, in that AZ the requested instance is not offered at all, no matter what maxPrice

apls777 commented 5 years ago

The formatting was fixed. Thanks.