search

pystorm / streamparse

Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.
http://streamparse.readthedocs.io/
Apache License 2.0
1.5k stars 218 forks source link

Can't run sparse run on wordcount example: RuntimeError: dictionary keys changed during iteration #487

Closed ghost closed 2 months ago

ghost commented 3 years ago

I'm using streamparse==3.16.0 and storm 1.1.3. When I try to run using python3.8.5 I get a RuntimeError with the following exception trace:

(VENV3) francis@francis-VirtualBox:~/Documents/repos/wordcount$ sparse run Traceback (most recent call last): File "/home/francis/Documents/repos/platform/VENV3/bin/sparse", line 8, in sys.exit(main()) File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/cli/sparse.py", line 87, in main args.func(args) File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/cli/run.py", line 127, in main run_local_topology( File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/cli/run.py", line 46, in run_local_topology topology_class = get_topology_from_file(topology_file) File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/util.py", line 573, in get_topology_from_file mod = importlib.import_module(mod_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "topologies/wordcount.py", line 11, in class WordCount(Topology): File "topologies/wordcount.py", line 13, in WordCount count_bolt = WordCountBolt.spec(inputs={word_spout: Grouping.fields("word")}, par=2) File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/storm/bolt.py", line 192, in spec return ShellBoltSpec( File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/dsl/bolt.py", line 23, in init super(ShellBoltSpec, self).init( File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/dsl/component.py", line 267, in init super(ShellComponentSpec, self).init( File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/dsl/component.py", line 41, in init self.inputs = self._sanitize_inputs(inputs) File "/home/francis/Documents/repos/platform/VENV3/lib/python3.8/site-packages/streamparse/dsl/component.py", line 84, in _sanitize_inputs for key, val in inputs.items(): RuntimeError: dictionary keys changed during iteration

vchrombie commented 3 years ago

Hi @frank-switchdin

The problem is you cannot iterate through a dictionary while it's changing during for loop. Casting the dictionary items to a list creates a list of its items, so you can iterate over it and avoid the RuntimeError.

I think this issue should be fixed with this commit 3368d61. You can try using the latest version 4.0.0 to avoid this error. Thanks.

kbourgoin commented 2 months ago

Cleaning out old issues. Please reopen if this is still an issue.