LourensVeen
commented
1 year ago Oh wow, I haven't seen this one before. The load balancer is receiving a message and tries to log that, and that log message is sent to the manager (I guess the remote log level is set to DEBUG?). That's a simple RPC call, and the manager should return a simple "everything okay" acknowledgement in return, but evidently there's extra data being sent and msgpack is getting confused over that.
I'll have a look at the source code and see if I can figure out how that could happen. Some kind of race condition sounds like a possibility, if somehow there's another thread communicating with the manager at the same time over the same socket. It could be interference with the profiling information being uploaded, but I thought I'd put some mutexes in there to keep this from happening. I'll get back to it.
DavidPCoster
After increasing the number of copies of my physics code from 10 to 20, I have started on seeing errors ...
Traceback (most recent call last): File "/afs/eufus.eu/user/g/g2dpc/GIT/ets_paf/UQ/load_balancer.py", line 60, in
main()
File "/afs/eufus.eu/user/g/g2dpc/GIT/ets_paf/UQ/load_balancer.py", line 34, in main
while instance.reuse_instance():
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/instance.py", line 232, in reuse_instance
do_reuse = self._decide_reuse_instance()
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/instance.py", line 851, in _decide_reuse_instance
got_f_init_messages = self._pre_receive()
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/instance.py", line 1018, in _pre_receive
self.pre_receive_f_init()
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/instance.py", line 1084, in __pre_receive_f_init
pre_receive(port_name, slot)
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/instance.py", line 1061, in pre_receive
msg, saved_until = self._communicator.receive_message(
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/communicator.py", line 386, in receive_message
_logger.debug('Received message on {}'.format(port_and_slot))
File "/gw/switm/python/3.10/lib/python3.10/logging/init.py", line 1465, in debug
self._log(DEBUG, msg, args, **kwargs)
File "/gw/switm/python/3.10/lib/python3.10/logging/init.py", line 1624, in _log
self.handle(record)
File "/gw/switm/python/3.10/lib/python3.10/logging/init.py", line 1634, in handle
self.callHandlers(record)
File "/gw/switm/python/3.10/lib/python3.10/logging/init.py", line 1696, in callHandlers
hdlr.handle(record)
File "/gw/switm/python/3.10/lib/python3.10/logging/init__.py", line 968, in handle
self.emit(record)
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/logging_handler.py", line 31, in emit
self._manager.submit_log_message(message)
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/mmp_client.py", line 137, in submit_log_message
self._call_manager(request)
File "/pfs/scratch/g2dpc/GIT/ets_paf/UQ/fusion_env/lib/python3.10/site-packages/libmuscle/mmp_client.py", line 285, in _call_manager
return msgpack.unpackb(response, raw=False)
File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
msgpack.exceptions.ExtraData: unpack(b) received extra data.
This is in a workflow consisting of an UQ component, a load balancer component, and then multiple copies of the physics code.
I have also seen a case that just hung where, I guess, everybody was waiting.
Is there a possible race condition?