search

astooke / Synkhronos

Extension to Theano for multi-GPU data parallelism
MIT License
20 stars 5 forks source link

Can only assign sequence of same size - SOLVED #5

Closed mharradon closed 7 years ago

mharradon commented 7 years ago

For posterity.

...
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/synkhronos/collectives.py", line 85, in all_reduce
    all_reduce_comm(gpu_vars, op, gpu=True)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/synkhronos/collectives.py", line 233, in all_reduce_comm
    arrays = collectives_prep(shared_vars, op=op)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/synkhronos/collectives.py", line 325, in collectives_prep
    sync.vars[:n_shared] = shared_IDs
ValueError: Can only assign sequence of same size

I was able to solve by calling synk.fork with a larger value for 'max_n_var' keyword argument - current default is 100 variables. Possibly could catch that exception and print a better error message? That one's a real head scratcher to look at - the code looks like this:

>>> size = len(x)
>>> y[:size] = x
ValueError: Can only assign sequence of same size
>>> # ¯\_(ツ)_/¯
astooke commented 7 years ago

Glad you found that setting...that was not a helpful error message haha.

On the to-do list is to re-work this part to communicate the needed information using the ZeroMQ component rather than pre-allocated multiprocessing arrays, so this won't even be a thing. It is leftover from before I had any of the ZeroMQ part.