Open vvivek921 opened 5 years ago
@vvivek921 I currently have the same problem. Did you find a way to load custom objects?
The KerasImageFileTransformer
is meant to be used with images. Try the KerasTransformer
class instead.
Example:
from sparkdl import KerasTransformer
from keras.models import Sequential
from keras.layers import Dense
import numpy as np
# Generate random input data
num_features = 10
num_examples = 100
input_data = [{"features" : np.random.randn(num_features).tolist()} for i in range(num_examples)]
input_df = sqlContext.createDataFrame(input_data)
# Create and save a single-hidden-layer Keras model for binary classification
# NOTE: In a typical workflow, we'd train the model before exporting it to disk,
# but we skip that step here for brevity
model = Sequential()
model.add(Dense(units=20, input_shape=[num_features], activation='relu'))
model.add(Dense(units=1, activation='sigmoid'))
model_path = "/tmp/simple-binary-classification"
model.save(model_path)
# Create transformer and apply it to our input data
transformer = KerasTransformer(inputCol="features", outputCol="predictions", modelFile=model_path)
final_df = transformer.transform(input_df)
Source: https://github.com/databricks/spark-deep-learning#working-with-images-in-spark
@ghunkins yep, I know. My problem is that for instance MobileNet is using custom objects like ReLU(6.) and DepthwiseConv2D, which can be loaded in Keras within a CustomObjectScope. However sparkdl's KerasImageFileTransformer seem not to have a parameter to load those custom objects.
@MrBanhBao You can edit the load model in KerasImageFileTransformer to accept a parameter to load custom objects and rebuild and use spark dl. I however haven't tried the above. I ended up not using spark DL.
Hi, I am using dice_loss custom object to train my model. Is there anyway to pass custom objects to load model in spark DL? Or is it that spark DL doesn't support loading models which have custom objects?
When I use keras to load the model, I am using
as mentioned here https://github.com/keras-team/keras/issues/3977
KerasImageFileTransformer doesn't support custom object loading. I am trying to run the below code which is failing.
The stack trace for failure is: