matterport / Mask_RCNN

Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow
Other
24.73k stars 11.71k forks source link

Save MaskRCNN full keras model #1299

Open oriolorra opened 5 years ago

oriolorra commented 5 years ago

Hi everyone!

I am trying to save the full model (not only the weight). I use the command:

_model.keras_model.save(pathfile) And it does not give any error.

But, when I try to load the full model shows an error, i am using these comands:

_from keras.models import loadmodel _model_test = load_model(pathfile)

And it shows this error:

_~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\models.py in load_model(filepath, custom_objects, compile) 268 raise ValueError('No model found in config file.') 269 model_config = json.loads(model_config.decode('utf-8')) --> 270 model = model_from_config(model_config, custom_objects=custom_objects) 271 272 # set weights

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\models.py in model_from_config(config, custom_objects) 345 'Maybe you meant to use ' 346 'Sequential.from_config(config)?') --> 347 return layer_module.deserialize(config, custom_objects=custom_objects) 348 349

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers__init__.py in deserialize(config, custom_objects) 53 module_objects=globs, 54 custom_objects=custom_objects, ---> 55 printable_module_name='layer')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name) 142 return cls.from_config(config['config'], 143 custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) + --> 144 list(custom_objects.items()))) 145 with CustomObjectScope(custom_objects): 146 return cls.from_config(config['config'])

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py in from_config(cls, config, custom_objects) 2523 # First, we create all layers and enqueue nodes to be processed 2524 for layer_data in config['layers']: -> 2525 process_layer(layer_data) 2526 # Then we process nodes in order of layer depth. 2527 # Nodes that cannot yet be processed (if the inbound node

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\engine\topology.py in process_layer(layer_data) 2509 2510 layer = deserialize_layer(layer_data, -> 2511 custom_objects=custom_objects) 2512 created_layers[layer_name] = layer 2513

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\layers__init__.py in deserialize(config, custom_objects) 53 module_objects=globs, 54 custom_objects=custom_objects, ---> 55 printable_module_name='layer')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\keras\utils\generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name) 136 if cls is None: 137 raise ValueError('Unknown ' + printable_module_name + --> 138 ': ' + class_name) 139 if hasattr(cls, 'from_config'): 140 custom_objects = custom_objects or {}

ValueError: Unknown layer: BatchNorm_

Has anyone tried to save a full model? Is it possible? has anyone a solution?

Thanks,

aashish-0393 commented 5 years ago

Please help if someone has done it before. @oriolorra did you find anything ? any progress?? please help

oriolorra commented 5 years ago

Hi @aashish-0393,

Yes, I have been able to save the full keras model. There is a webpage inside keras wiki that explains how to create a custom layer. All you have to do is to write some more methods inside the custom layers of MaskRCNN.

See you

NorthLatitudeOne commented 5 years ago

@oriolorra could you share your sample code?

PhanDuc commented 5 years ago

You can make some modifications in the model.py. There will be keras.callbacks.ModelCheckpoint, where you can choose to save weight or full model.

oriolorra commented 5 years ago

Hi @NorthLatitudeOne,

Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras.

Best regards, model.py.txt

shivasj commented 5 years ago

I was able to save the model using the following code:

import keras
# Save the Keras Model
keras.models.save_model(model.keras_model,"mask_rcnn_coco.hdf5")

# Save weights
model.keras_model.save_weights("mask_rcnn_coco.h5")

# Save model to JSON
model_json = model.keras_model.to_json()
with open("mask_rcnn_coco.json", "w") as json_file:
    json_file.write(model_json)

But when i try to load the saved model, i get the following error:

from mrcnn.model import BatchNorm
model2 = keras.models.load_model("mask_rcnn_coco.hdf5",custom_objects={"BatchNorm": BatchNorm})

~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in call(self, inputs, mask) 685 if has_arg(self.function, 'mask'): 686 arguments['mask'] = mask --> 687 return self.function(inputs, **arguments) 688 689 def compute_mask(self, inputs, mask=None): ~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in (t) 855 # Reshape to [batch, anchors, 2] 856 rpn_class_logits = KL.Lambda( --> 857 lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x) 858 859 # Softmax on last dimension of BG/FG. NameError: name 'tf' is not defined

Any idea on how i can make "tf" available in the Lambda function?

K-M-Ibrahim-Khalilullah commented 5 years ago

@oriolorra

Would you tell me please, where you changed in model.py? If I use your shared model.py file and use model.keras_model.save(dirPath), then is it OK?

K-M-Ibrahim-Khalilullah commented 5 years ago

@oriolorra Would you tell me please, where you changed in model.py? If I use your shared model.py file and use model.keras_model.save(dirPath), then is it OK? @oriolorra would you share your config and inferenceconfig data please. Beacause when I am going to run your model, it gives some error: such as 'InferenceConfig' object has no attribute 'TRAIN_BN' Also for config.IMAGE_META_SIZE I am waiting for your response. thanks

K-M-Ibrahim-Khalilullah commented 5 years ago

@oriolorra I run your model.py file to save full model. The code is here: ` with tf.device(DEVICE): model = modellib.MaskRCNN(mode="training", config=config, model_dir=MODEL_DIR)

init_with = "coco"

if init_with == "imagenet": model.load_weights(model.get_imagenet_weights(), by_name=True) elif init_with == "coco":
print("Loading weights ", COCO_MODEL_PATH) model.load_weights(COCO_MODEL_PATH, by_name=True) " elif init_with == "last":
model.load_weights(model.find_last()[1], by_name=True)

model.train(dataset_train, dataset_val, learning_rate=config.LEARNING_RATE, epochs=2, layers="all")

model_path = os.path.join(MODEL_DIR, "mask_rcnn1.h5") model_path2 = os.path.join(MODEL_DIR, "mask_rcnn2.h5") model.keras_model.save_weights(model_path) model.keras_model.save(model_path2) `

but it gives the following error:

TypeError: can't pickle _thread.RLock objects

How can I save full model?

denizsimsek06 commented 5 years ago

@ibrahimLearning Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/ You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py.

Then when loading I used

new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

K-M-Ibrahim-Khalilullah commented 4 years ago

@oriolorra I used your model.py to save full model. But when I am going to save using model.keras_model.save(model_path; it gives the following error: TypeError: can't pickle _thread.RLock objects

Is there any suggestion please?

Thanks

oriolorra commented 4 years ago

@ibrahimLearning, I do not which problem is this one.

Sorry

suchiz commented 4 years ago

For anyone who comes here again ... I've done a work around. I was unable to save directly the model after training, like mostly everybody. So what I've done was to save weights only during the training and then during the evaluate or the splash, I load the inference model with my trained weights like usually and I just call a model.keras_model.to_json() and then save the json after the model.detect. Hope it will work for you too ...

gabola24 commented 4 years ago

Hi @NorthLatitudeOne,

Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras.

Best regards, model.py.txt

Hi @oriolorra, thanks for your help. Your fixes in model.py it manage to solve the Batchnorm layer export. But now im looking the way to manage the other layers by the same way. Did you export the model without pass the custom objects " ProposalLayer,PyramidROIAlign,DetectionLayer" ?

vagomundo21 commented 4 years ago

Hi @NorthLatitudeOne, Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras. Best regards, model.py.txt

Hi @oriolorra, thanks for your help. Your fixes in model.py it manage to solve the Batchnorm layer export. But now im looking the way to manage the other layers by the same way. Did you export the model without pass the custom objects " ProposalLayer,PyramidROIAlign,DetectionLayer" ?

Hey, I might have solution for this. I faced similar issue. Here, I've given some info on how to save the model.

bilalahmed381 commented 4 years ago

@ibrahimLearning Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/ You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py.

Then when loading I used

new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

I tried this and got this error TypeError: init() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'

vagomundo21 commented 4 years ago

hi, I have a problem, seems like no one work on it.

Mask-RCNN on video to remove background issues

I'm done with image segmentation on image/video, now I didn't find any clue to remove the background of this segmented video. In matterport Repo ----> ( https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/Mask_RCNN.ipynb) they use the following code to remove the segmented image background------>

`def segment(image, r): idx = r['scores'].argmax() mask = r['masks'][:,:,idx] mask = np.stack((mask,)_3, axis=-1) mask = mask.astype('uint8') bg = 255 - mask * 255 mask_img = image_mask result = mask_img+ bg return result

segmentation = segment(image, r) plt.subplots(1, figsize=(16, 16)) plt.axis('off') plt.imshow(np.concatenate([image, segmentation], axis = 1))`

How can I use the same for the video to remove the background? https://colab.research.google.com/drive/1akpTRMrebgCCvFV_B-G9ADfhEaoPnUSY#scrollTo=md3eCFXQbGue

Because it works kind of same way --> it converts the Video to images frames. And then its Process images, in last step its Turn processed images to output videos.

So, does removing background frame by frame of a video not work? Does it work on single image?

vagomundo21 commented 4 years ago

@ibrahimLearning Look at this guide: https://keras.io/layers/writing-your-own-keras-layers/ You have to add "build", "compute_output_shape", and "get_config" methods to the custom layers implemented in model.py. Then when loading I used new_model = tf.keras.models.load_model('convert_1.h5', custom_objects={'ProposalLayer': modellib.ProposalLayer, #add dict entries like above for each custom layer})

I tried this and got this error TypeError: init() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'

It happens because those parameters are dynamically defined in model.py; I've explicitly declared those variables, have a look here.

harshays98 commented 3 years ago

Hi @NorthLatitudeOne, Sorry for the delay. Here you have the model.py file. With this, I was able to save the full model with Keras. Best regards, model.py.txt

Can you please mention what changes you have done in this file?

Can we save full model by giving save_weights_only=False?

AndreyStille commented 3 years ago

from mrcnn.model import BatchNorm

Hi. Have you solved the issue with NameError: name 'tf' is not defined? I have same error. Please help, if you can.

garnetcode commented 3 years ago

I was able to save the model using the following code:

import keras
# Save the Keras Model
keras.models.save_model(model.keras_model,"mask_rcnn_coco.hdf5")

# Save weights
model.keras_model.save_weights("mask_rcnn_coco.h5")

# Save model to JSON
model_json = model.keras_model.to_json()
with open("mask_rcnn_coco.json", "w") as json_file:
    json_file.write(model_json)

But when i try to load the saved model, i get the following error:

from mrcnn.model import BatchNorm
model2 = keras.models.load_model("mask_rcnn_coco.hdf5",custom_objects={"BatchNorm": BatchNorm})

~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in call(self, inputs, mask) 685 if has_arg(self.function, 'mask'): 686 arguments['mask'] = mask --> 687 return self.function(inputs, **arguments) 688 689 def compute_mask(self, inputs, mask=None): ~/anaconda/envs/bc3/lib/python3.6/site-packages/keras/layers/core.py in (t) 855 # Reshape to [batch, anchors, 2] 856 rpn_class_logits = KL.Lambda( --> 857 lambda t: tf.reshape(t, [tf.shape(t)[0], -1, 2]))(x) 858 859 # Softmax on last dimension of BG/FG. NameError: name 'tf' is not defined

Any idea on how i can make "tf" available in the Lambda function?

include 'tf': tf in custom objects ie

import tensorflow as tf

custom_objects={'tf': tf}

OUTHIM commented 3 years ago

I find the method which might help you. Link: https://github.com/matterport/Mask_RCNN/issues/1226#issuecomment-859212472

novaHardware commented 3 years ago

For anyone who comes here again ... I've done a work around. I was unable to save directly the model after training, like mostly everybody. So what I've done was to save weights only during the training and then during the evaluate or the splash, I load the inference model with my trained weights like usually and I just call a model.keras_model.to_json() and then save the json after the model.detect. Hope it will work for you too ...

Hi @suchiz, so did you modified model.py as @vagomundo21 proposed? and then you used model.keras_model.to_json()? thanks