excamera / mu

Framework to Run General-Purpose Parallel Computations on AWS Lambda
94 stars 23 forks source link

Add option to specify output s3 bucket #4

Open shelbyt opened 8 years ago

shelbyt commented 8 years ago

Output s3 buckets are hardcoded based on the name of the input parameters: outName = "%s-y4m_%02d" % (ServerInfo.video_name, ServerInfo.num_frames)

ServerInfo should have another option to specify an output s3 bucket, else use the default parameters.

kwantam commented 8 years ago

ServerInfo doesn't really have much to do with core mu functionality. libmu/server.py has some convenience functions that, e.g., construct help messages and getopt strings, but probably it's better to break those out from mu rather than try to bake more and more options into them.

In other words, it's probably better to think of ServerInfo as something the user implements, not something mu provides. That we're doing the latter right now is a result of laziness/lack of time while we were pushing toward the submission.

shelbyt commented 8 years ago

Looking at the code it looks like there are two classes of configurations:

1) Options that are needed by Mu to spawn lambdas (i.e. regions, lambda_function, cacert) 2) Options that the user wants to keep track of for his own convenience (i.e. video_name, in_format?)

Currently both of these types of options are tightly coupled in as a ServerInfo object and used in libmu/server.py but we want to break these out of Mu. Does that sound about correct?

kwantam commented 8 years ago

Yup, exactly.

shelbyt commented 8 years ago

This is probably just a terminology issue but it's not clear to me what you mean by breaking them out of Mu if Mu needs Class 1 options to function.

My initial thought was to have a matching .cfg file for each server with just Class 1 options that can be validated in a checking stage like you've mentioned in #3.

Then as usual the user can do whatever they want with their Class 2 options.

kwantam commented 8 years ago

Sorry, I should have clarified in my previous answer that "breaking out" should only apply to class 2, as you say.