search

conveyal / analysis-backend

Server component of Conveyal Analysis
http://conveyal.com/analysis
MIT License
23 stars 12 forks source link

Boost regional analyses #192

Closed ansoncfit closed 5 years ago

ansoncfit commented 5 years ago

With #191 and related changes (e.g. per-second EC2 billing), it should now be fairly safe to allow end users to launch spot instances.

In the UI, maybe this would be a 🚀 to the left of the regional analysis status update bar? We could start with a low limit (e.g. 20) on user-requested spot instances.

We could also consider starting multiple workers automatically when a regional analysis is launched.

ansoncfit commented 5 years ago

As part of this task, I'm refactoring much of the EC2-specific code out of Broker. I plan to use EC2Fleet to take advantage of a single interface for both spot and on-demand instance requests. I'm also considering having the regional analysis controller launch ~50 spot instances after a few tasks have successfully been completed on a regional analysis.

ansoncfit commented 5 years ago

It appears that the launch template overrides for EC2 Fleet do not yet support user data, which we use to provide the worker startup script.

Currently, the only supported override is instance type.

So back to separate on-demand and spot requests.

ansoncfit commented 5 years ago

Closed by #201, merged into master in #204