Closed PankeshGupta closed 6 years ago
@PankeshGupta can you give me some indication on how you run elephas and your full stack trace? Thanks.
Looks like you're using Keras constraints and elephas isn't able to de/serialise them properly. Can't know for sure at this point.
I am also getting this error in 'synchronous' mode:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-14-63c8e305d765> in <module>()
1 from pyspark.mllib.evaluation import MulticlassMetrics
2
----> 3 fitted_pipeline = pipeline.fit(train_df) # Fit model to data
4
5 prediction = fitted_pipeline.transform(train_df) # Evaluate on train data.
~/spark/python/pyspark/ml/pipeline.py in fit(self, dataset, params)
67 return self.copy(params)._fit(dataset)
68 else:
---> 69 return self._fit(dataset)
70 else:
71 raise ValueError("Params must be either a param map or a list/tuple of param maps, "
~/spark/python/pyspark/ml/pipeline.py in _fit(self, dataset)
211 dataset = stage.transform(dataset)
212 else: # must be an Estimator
--> 213 model = stage.fit(dataset)
214 transformers.append(model)
215 if i < indexOfLastEstimator:
~/spark/python/pyspark/ml/pipeline.py in fit(self, dataset, params)
67 return self.copy(params)._fit(dataset)
68 else:
---> 69 return self._fit(dataset)
70 else:
71 raise ValueError("Params must be either a param map or a list/tuple of param maps, "
~/anaconda/envs/spacy/lib/python3.6/site-packages/elephas-0.3-py3.6.egg/elephas/ml_model.py in _fit(self, df)
57 num_workers=self.get_num_workers())
58 spark_model.train(simple_rdd, nb_epoch=self.get_nb_epoch(), batch_size=self.get_batch_size(),
---> 59 verbose=self.get_verbosity(), validation_split=self.get_validation_split())
60
61 model_weights = spark_model.master_network.get_weights()
~/anaconda/envs/spacy/lib/python3.6/site-packages/elephas-0.3-py3.6.egg/elephas/spark_model.py in train(self, rdd, nb_epoch, batch_size, verbose, validation_split)
192
193 if self.mode in ['asynchronous', 'synchronous', 'hogwild']:
--> 194 self._train(rdd, nb_epoch, batch_size, verbose, validation_split, master_url)
195 else:
196 print("""Choose from one of the modes: asynchronous, synchronous or hogwild""")
~/anaconda/envs/spacy/lib/python3.6/site-packages/elephas-0.3-py3.6.egg/elephas/spark_model.py in _train(self, rdd, nb_epoch, batch_size, verbose, validation_split, master_url)
224 new_parameters = self.master_network.get_weights()
225 for delta in deltas:
--> 226 constraints = self.master_network.constraints
227 new_parameters = self.optimizer.get_updates(self.weights, constraints, delta)
228 self.master_network.set_weights(new_parameters)
AttributeError: 'Sequential' object has no attribute 'constraints'
@dbl001 it seems at one point in time you could access constraints
in Keras directly from models, e.g. here by calling model.constraints
. At some point this must have stopped working. Would you mind helping out by checking where accessing constraints went? this could be a neat little PR.
Investigating …
On May 10, 2018, at 11:07 AM, Max Pumperla notifications@github.com wrote:
@dbl001 https://github.com/dbl001 it seems at one point in time you could access constraints in Keras directly from models, e.g. here by calling model.constraints. At some point this must have stopped working. Would you mind helping out by checking where accessing constraints went? this could be a neat little PR.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maxpumperla/elephas/issues/87#issuecomment-388136421, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9i2yUhQ_f5SHojYL4eGTkPFpUYLEUlks5txIH4gaJpZM4Tt0ut.
Have you seen this discussion? It show how Constraints are currently implemented in Keras.
https://github.com/keras-team/keras/issues/8196
You can call get_config() in constraints.py to get the value of max_norm() e.g.
def get_config(self): return {'max_value': self.max_value, 'axis': self.axis}
You can also define custom constraints.
Will this approach e.g - getting the default or custom constrain asdefined in keras.constraint work in the Elephas optimizers.py function get_updates()?
def get_updates(self, params, constraints, grads): lr = self.lr (1.0 / (1.0 + self.decay self.iterations)) self.updates = [(self.iterations, self.iterations + 1.)] new_weights = []
for p, g, c in zip(params, grads, constraints):
m = np.zeros_like(p) # momentum
v = self.momentum * m - lr * g # velocity
if self.nesterov:
new_p = p + self.momentum * v - lr * g
else:
new_p = p + v
new_weights.append(c(new_p))
return new_weights
In any case, constraints = self.master_network.constraints
won’t work.
On May 10, 2018, at 11:07 AM, Max Pumperla notifications@github.com wrote:
@dbl001 https://github.com/dbl001 it seems at one point in time you could access constraints in Keras directly from models, e.g. here by calling model.constraints. At some point this must have stopped working. Would you mind helping out by checking where accessing constraints went? this could be a neat little PR.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maxpumperla/elephas/issues/87#issuecomment-388136421, or mute the thread https://github.com/notifications/unsubscribe-auth/AC9i2yUhQ_f5SHojYL4eGTkPFpUYLEUlks5txIH4gaJpZM4Tt0ut.
this is now fixed on master.
I get this error while training a keras model .. keras : Keras 2.1.6
elephas : latest updated from master branch