search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Bootstrap of bolts/spouts environment through scripts #31

Open poros opened 9 years ago

poros commented 9 years ago

@anusha-r:

MRJob allows a bunch of options for users to customize the execution environment - eg: users can specify commands to be executed at startup, set env variables, upload python archives that have to be added to PYTHONPATH and so on.

For the hadoop environment, MRJob simply invokes an autogenerated script that performs all these tasks that the user specifies, and then invokes the actual python module to run. Would it be possible to support something similar in Pyleus? For instance, the user could specify a shell script or some command to run at startup, before the python module is invoked. This will ensure changes to python path happen before we enter python land.

I have a partial workaround now, to execute a setup script during bolt initialization using call(..) API, but having pyleus let me customize the way my bolt module is invoked would be a better fix.

Moved from PYLEUS-86

darionyaphet commented 9 years ago

could setting python path in config file ? Or make a pre-load script

patricklucas commented 9 years ago

@anusha-r: What extra python modules would you need that can't be included in the virtualenv?

I guess a better question is, is this an optimization, or something you can't otherwise do with requirements.txt?