search

rossumai / keras-multi-gpu

Multi-GPU data-parallel training in Keras
MIT License
77 stars 20 forks source link

tensorflow : 'NoneType' object has no attribute 'update' #8

Open Nqabz opened 6 years ago

Nqabz commented 6 years ago

I tried running all your example on 8 Nvidia V100 however I get this error across all of them:

  File "/opt/conda/lib/python3.5/copy.py", line 182, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/conda/lib/python3.5/copy.py", line 297, in _reconstruct
    state = deepcopy(state, memo)
  File "/opt/conda/lib/python3.5/copy.py", line 155, in deepcopy
    y = copier(x, memo)
  File "/opt/conda/lib/python3.5/copy.py", line 243, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/opt/conda/lib/python3.5/copy.py", line 182, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/opt/conda/lib/python3.5/copy.py", line 306, in _reconstruct
    y.__dict__.update(state)
AttributeError: 'NoneType' object has no attribute 'update'

This is in a docker environment with the following versions: **

** Could this be version compatibility or GPU device compatibility? Thanks for any pointers.

DNXie commented 6 years ago

https://github.com/keras-team/keras/issues/8649

cagbal commented 6 years ago

Same issue here! 

Tensorflow 1.4.1
Keras 2.1.5
Python 3.5.5
CUDA 8.0
bzamecnik commented 5 years ago

This issue also appears to me when I have two models sharing layers - one for prediction, one for training (with CTC loss and auxiliary inputs). ModelCheckpoint is saving from the training model.