AWS automation - Githubissues

e-koch commented 8 years ago

Scripts and alterations to work with the aws_controller package.

@tetarenk - If you don't have any more changes, I'll merge this in.

tetarenk commented 8 years ago

Yep, go ahead, I have no more changes.

e-koch commented 8 years ago

Success!

tetarenk commented 8 years ago

Awesome! Nice work!

Alex Tetarenko PhD Student Department of Physics University of Alberta Edmonton, Canada http://www.ualberta.ca/~tetarenk

On Jan 7, 2016, at 7:45 PM, Eric Koch notifications@github.com wrote:

Success!

— Reply to this email directly or view it on GitHub.

e-koch commented 8 years ago

@tetarenk - Here's a list of the changes I made to the parameters, etc...

runClean is a string like the rest of the other optional parameters. This is just easier for the json read-in.
Parameters are read in from a JSON saved text file. This makes it much easier since messages are sent to the worker instance from through the AWS SQS messaging service. I wrote a conversion function in utils.py that converts the example in param.txt into a JSON format (or a dictionary to give to run).
run in aws_exec.py runs the whole process. It requires some inputs which will be provided by the web-server. For now, it is setup through running test_aws_exec.py. The upload and download locations will have to be changed. Also it defaults to using an m3.large instance, which is adequate for test runs. The maximum time allowed for the process to run is 1 day right now (line 80).
I added a few error catches in the analysis scripts so the process will exit quicker when there's a problem. The web server will have a parameter check so computing time isn't wasted.

Astroua / AstroCompute_Scripts