Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Runtime configuration settings #105

Closed lsvx closed 9 years ago

lsvx commented 9 years ago

I would like to use configuration/settings files for different environments. For example, my bolt makes a GET request to localhost on my QA environment, but in production it should hit xxx.xxx.xxx.xxx.

Is there any way to pass arbitrary variables during the JAR build (like Storm's storm jar my-topology.jar MyTopology -c environment=qa) so that my components can identify the the correct configuration file to import? Is there a different way to accomplish this goal?

Should I simply read OS environment variables?

import os

os.environ.get['ENV']

Thanks, Lucas

poros commented 9 years ago

The -c option of the command line is just for specifying a path for your Pyleus config file, you can't pass arguments using that.

Using OS environment variables is definitely an option here (I assume you have different Storm clusters for your test and production environment).

If you don't have the possibility to set environment variables, you may achieve the same result passing different options to bolt basing on your environment. Unfortunately, since we do not have a "real" Python API for topology definition, you'll need different YAML files for environment. Since this is basically code duplication, I would go for the environment variables option.

lsvx commented 9 years ago

Right. This is what I figured. Ok thank you very much for your input!