pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

Handle when components are defined inside topology module #441

Open dan-blanchard opened 6 years ago

dan-blanchard commented 6 years ago

I recently rediscovered that you cannot define a Bolt or Spout inside a topology module, because the topology module does not get packaged up with the the rest of the code. The error message you get is really cryptic, because the Storm workers will crash when launching the process telling you the topology module cannot be found. We should either explicitly detect and disallow this behavior, or add the topology file to the JAR that gets created. (Personally, I think adding the Python file to the JAR would be difficult, because the import paths in the topology module itself are unlikely to still work.)