When initializing SparkAsyncDL with optimizer='adam', we get the following exception
Traceback (most recent call last):
File "spark-train.py", line 170, in
p = Pipeline(stages=[va, spark_model]).fit(train)
File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit
return self._fit(dataset)
File "/usr/local/spark/python/pyspark/ml/pipeline.py", line 109, in _fit
model = stage.fit(dataset)
File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit
return self._fit(dataset)
File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 278, in _fit
tf_optimizer = build_optimizer(self.getTfOptimizer(), self.getTfLearningRate(), optimizer_options)
tensorflow.version = 2.3.0
sparkflow = 0.7.0
File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 20, in build_optimizer
'adam': tf.train.AdamOptimizer,
AttributeError: module 'tensorflow._api.v2.train' has no attribute 'AdamOptimizer'
When initializing SparkAsyncDL with optimizer='adam', we get the following exception Traceback (most recent call last): File "spark-train.py", line 170, in
p = Pipeline(stages=[va, spark_model]).fit(train)
File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit
return self._fit(dataset)
File "/usr/local/spark/python/pyspark/ml/pipeline.py", line 109, in _fit
model = stage.fit(dataset)
File "/usr/local/spark/python/pyspark/ml/base.py", line 132, in fit
return self._fit(dataset)
File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 278, in _fit
tf_optimizer = build_optimizer(self.getTfOptimizer(), self.getTfLearningRate(), optimizer_options)
tensorflow.version = 2.3.0 sparkflow = 0.7.0 File "/data/#user_folder#/sparkflow/sparkflow/tensorflow_async.py", line 20, in build_optimizer 'adam': tf.train.AdamOptimizer, AttributeError: module 'tensorflow._api.v2.train' has no attribute 'AdamOptimizer'