search

ch3njust1n / smpl

Simultaneous Multi-Party Learning Framework
0 stars 0 forks source link

IndexError: string index out of range #91

Closed ch3njust1n closed 6 years ago

ch3njust1n commented 6 years ago 
Process Process-4:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda2/envs/smpl/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
INFO:root:('192.168.0.11', 49402) closed socket
    self.run()
  File "/home/ubuntu/anaconda2/envs/smpl/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/smpl/distributed/parameter_server.py", line 304, in __train_hyperedge
    sess_id = self.__init_session(log=log, log_path=log_path)
  File "/home/ubuntu/smpl/distributed/parameter_server.py", line 615, in __init_session
INFO:root:reconnecting to ('192.168.0.11', 49402)
    self.cache.set(sess_id, ujson.dumps({"parameters": model[0], "accuracy": model[1], "val_size": 0,
IndexError: string index out of range
ch3njust1n commented 6 years ago

Something to do with broken pipes which only occurs once in a while probably because of race conditions. Hard to say, but

  1. Place if conditions around all pc.send() calls to handle broken pipes and invalid responses
  2. pc.send() should also gracefully handle all None or empty content so that caller doesn't break unexpectedly