search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
404 stars 113 forks source link

Pyleus ignoring virtualenv #110

Open jwestboston opened 9 years ago

jwestboston commented 9 years ago

We are running the latest Pyleus development branch on CentOS 7 hosts.

For some reason, spouts and bolts are not honoring the virtualenv. We cannot import a module included from requirements.txt within a spout or bolt. Yet, the module is indeed shipped within the virtualenv inside the .jar file.

Definitely not buildling with the -s flag or instructing it to use system packages.

Any advice on how to debug this? Where is the code which actually calls/loads/activate the virtualenv, when launching the spout or bolt?

poros commented 9 years ago

The virtualenv isn't activated, we simply call its Python interpreter. You can find the code here.

This is probably the same thing happening in issue #29. Unfortunately, I am still not able to figure out the reason behind this issue :( Any help would be really appreciated.

jwestboston commented 9 years ago

@poros Ahh ok interesting. What am I doing wrong, then? How should I have Pyleus properly activate the virtualenv? Or is this not a feature yet supported?

poros commented 9 years ago

You're doing nothing wrong (at least I guess, I didn't see your code). Activating a virtualenv after moving it wasn't a trivial task to achieve and we found out that just calling the interpreter inside was enough for it to figure out where to look for dependencies.

This (or something else we still don't know) seems to lead to bugs on some distros/environments/configurations/systems. There are still a lot of unknowns in this issue.

jwestboston commented 9 years ago

@poros Interesting and thanks for the clarification/update. For now, I will install the python packages in requirements.txt system-wide.