Traceback (most recent call last):
File "/users/tareq/Proteus/examples/megatron_gpt.py", line 494, in <module>
cost = sim.run('log/trace')
File "/users/tareq/Proteus/proteus/simulator/simulator.py", line 1077, in run
cur_stage = dev_group.execute()
File "/users/tareq/Proteus/proteus/simulator/simulator.py", line 781, in execute
self.dev_execute(cur_stage)
File "/users/tareq/Proteus/proteus/simulator/simulator.py", line 811, in dev_execute
ntsks = self.executors[tsk.rank].alloc(tsk)
KeyError: 0
Upon further digging, I found that the simulator is allocating a task tsk = f7806[x_2_148/0:gpu:0] to executors while self.executors has the value:
self.executors = {
6: <proteus.simulator.simulator.DevExecutor object at 0x7f80dd2f3220>,
7: <proteus.simulator.simulator.DevExecutor object at 0x7f80dd2f32b0>,
5: <proteus.simulator.simulator.DevExecutor object at 0x7f80dd2f3190>,
4: <proteus.simulator.simulator.DevExecutor object at 0x7f80dd2f3100>
}
The rank of the task is not present in the executor dictionary, hence throwing a KeyError. Am I doing this wrong?
I get a
KeyError
after running the following command.The error:
Upon further digging, I found that the simulator is allocating a task
tsk = f7806[x_2_148/0:gpu:0]
to executors whileself.executors
has the value:The rank of the task is not present in the executor dictionary, hence throwing a
KeyError
. Am I doing this wrong?