search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

Consider using shiv instead of virtualenvs for deployment #445

Open dan-blanchard opened 6 years ago

dan-blanchard commented 6 years ago

Shiv is much lighter weight than PEX, which we had previously discussed using in #212. It just uses pip's underlying mechanisms for packaging things up in zipapp, so I'd prefer to use it.

You can see how to use it for cross-platform building in this comment.

kbourgoin commented 6 years ago

This would be amazing. A big :+1: from me on getting this working. It'd massively simplify deploying to large clusters, not to mention resizing clusters on the fly.

dan-blanchard commented 5 years ago

Hit an issue with only being able to pass a single command to ShellComponent via Thrift, so I filed STORM-3299. STORM-152 is also annoying, but if I could pass multiple arguments to a script, I could work around it easily.