search

Yelp / pyleus

Pyleus is a Python framework for developing and launching Storm topologies.
Apache License 2.0
403 stars 107 forks source link

Added --always-copy to virtualenv call #55

Open justquick opened 9 years ago

justquick commented 9 years ago

This make it so that the files in the vent are copied from the source machine when creating the jar files. If this is not present, recent versions of virtualenv will symlink and cause bad links in the jar files. Bad links mean that the files will not exist when the jar is actually run. This fix ensures that all required files in the venv are copied over so you do not actually have to have python installed on your pyleus cluster.

This is the root cause of the ImportError: No module named 'encodings' problem in #50

patricklucas commented 9 years ago

How does this affect the size of a Pyleus jar?

justquick commented 9 years ago

in testing the example exclamation_topology it's an increase of ~15% (25MB vs 30MB)

patricklucas commented 9 years ago

This is something I'd like to bring in, but I'm concerned about JAR bloat—I'm going to open another ticket regarding the removal of the scala/ and kafka/ dirs from the JAR if the user is not using a Kafka spout. I would prefer to have that change in the same release or earlier as this one.

patricklucas commented 9 years ago

Opened #57.

justquick commented 9 years ago

@patricklucas I could add a "--always-copy" flag to pyleus build which would pass that flag onto virtualenv when initing the environment. it could b optional since only in specific cases (eg python3.4) is this absolutely required to get the stdlib installed and functioning properly. we can trust that users are smart enough to use this flag when running pyleus build only when they absolutely need to.