search

radical-cybertools / radical.pilot

RADICAL-Pilot
http://radical-cybertools.github.io/radical-pilot/index.html
Other
54 stars 23 forks source link

Error during the pilot update procedure #3227

Open mtitov opened 2 weeks ago

mtitov commented 2 weeks ago

Error message from PMGR logs

1725478545.765 : pmgr.0000            : 24356 : 140146504709888 : DEBUG    : call <bound method Pilot._default_state_cb of <Pilot object, uid pilot.0000>>
1725478545.765 : pmgr.0000            : 24356 : 140146504709888 : DEBUG    : pilot.0000 calls cb <bound method Pilot._default_state_cb of <Pilot object, uid pilot.0000>>
1725478545.766 : pmgr.0000            : 24356 : 140146504709888 : INFO     : [Callback]: pilot pilot.0000 state: PMGR_LAUNCHING_PENDING.
1725478545.768 : pmgr.0000            : 24356 : 140146504709888 : ERROR    : callback error
Traceback (most recent call last):
  File "/home/ja961/anaconda3/envs/pyr/lib/python3.10/site-packages/radical/utils/zmq/pubsub.py", line 304, in _listener
    cb(topic, msg)
  File "/home/ja961/anaconda3/envs/pyr/lib/python3.10/site-packages/radical/pilot/pilot_manager.py", line 307, in _state_sub_cb
    if not self._update_pilot(thing, publish=False):
  File "/home/ja961/anaconda3/envs/pyr/lib/python3.10/site-packages/radical/pilot/pilot_manager.py", line 355, in _update_pilot
    self._pilots[pid]._update(pilot_dict)
  File "/home/ja961/anaconda3/envs/pyr/lib/python3.10/site-packages/radical/pilot/pilot.py", line 260, in _update
    for _,cb_val in self._callbacks[rpc.PILOT_STATE].items():
RuntimeError: dictionary changed size during iteration

RADICAL stack

  radical.gtod         : 1.61.0
  radical.pilot        : 1.62.0
  radical.utils        : 1.61.0