search

cerndb / dist-keras

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
http://joerihermans.com/work/distributed-keras/
GNU General Public License v3.0
623 stars 169 forks source link

Strange "stateful" behavior #50

Closed marty90 closed 6 years ago

marty90 commented 6 years ago

I'm writing a simple model for Dist-Keras. If I change some parameter of the Neural Network (for example the size of some layer), I get errors like: ValueError: Cannot feed value of shape (16, 128) for Tensor u'Placeholder_1:0', which has shape '(15, 128)' In fact, I changed a layer size from 16 to 15, but it seems like the system has memory of the old size and prompts this error.

marty90 commented 6 years ago

I found the problem: there was an old dist-kers instance stuck on port 5000, and it contained the old version of the model. So I wasn't able to update my code.