search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

How do I bundle a static file as a resource #353

Closed kbruegge closed 7 years ago

kbruegge commented 7 years ago

Hello everyone,

lets say I have a bolt which requires some input file to do some computations. This file never changes and is quite small. Imagine a use case where I want to use some pickled scikit-learn model to perform a prediction. Adding the file to the :resource-paths ["_resources"] folder does not seem to work. It gets bundled in the Jar I believe. However I don't know how to access it from python then. Any pointers would be much appreciated.

dan-blanchard commented 7 years ago

The simplest method for shipping that along is to just put the file in the same directory as your code before you do sparse submit, that'll make it get put in the JAR and have it get uncompressed in the same directory as your Python code. Then you can just do the usual os.path.dirname(os.path.realpath(__file__)) trick to find the current directory and load your model.

The _resources folder is a special folder that streamparse creates and destroys at will, so you can't rely on anything actually being in there if you put it in there.