dbalv
commented
5 years ago I have the same problem too. There is also a previous issue on this theme. https://github.com/matterport/Mask_RCNN/issues/308
Open roburst2 opened 5 years ago
dbalv
commented
5 years ago I have the same problem too. There is also a previous issue on this theme. https://github.com/matterport/Mask_RCNN/issues/308
roburst2
commented
5 years ago @dbalv is this working for you Issue resolved: changing 'save_weights_only=True" to "best_model_only = True" caused this error.
dbalv
commented
5 years ago I could save the models in these configs:
Without problems.
But i would prefer to save the full model in this config since i need to resume training for an application i am developing:
95xueqian
commented
5 years ago have you solved this problem？I have the same problem with you. @roburst2
roburst2
commented
5 years ago No @95xueqian
ShujianHu
commented
5 years ago @roburst2 Lambda functions cannot be pickled and there are layers which are clearly self-defined. I tried to pull save_model function out from keras.model and re-run the saving process and there existed a get_config function needed. Keras contributor said it would be safer to rewrite the get_config function for custom layers and I'm struggling on that.
95xueqian
commented
5 years ago roburst2
commented
5 years ago @ShujianHu i changed all lambda functions with normal function, still i am facing the error
roburst2
commented
5 years ago @95xueqian there is no issue on that link
ShujianHu
commented
5 years ago @roburst2 I believe it's pickle function itself which cannot deal with lambda and return things like recursion error with deepcopy. And it's recommended by Keras to json-serialize the model, not to pickle it. I think model.save() cannot serialize the model is a completely other issue, look close to model and I found there layers class inherited from original layers defined in keras which didn't provide encode and decode function (get_config &from_config) in it and I guess that's the problem.
nlslzf
commented
5 years ago @ShujianHu i want to run maskrcnn on elephas,and it used to_json() of keras model,and get the same error. any one solved this problem?
zakiahmedkgp
commented
5 years ago @roburst2 I am still having this problem in saving the complete architecture. I have also tried saving architecture and weights separately but the problem still exists. https://github.com/matterport/Mask_RCNN/issues/918#issuecomment-501148906
In network.py i replaced "return copy.deepcopy(config) " to "return config" but got new error json is not serializable. I tried to change all lambda with a function but still getting the can't pickle error. I tried to save the model before training, getting same error. i tried below code as well to save the model _model.save('mask_rcnn.h5').
with open('model_architecture_subin.json', 'w') as f: f.write(model.tojson()) with yaml also i tried but same issue.
since the problem might be in lambda function can you tell how to change the lambda functions in the model structure. Thanks in advance.
lukeb94
commented
5 years ago Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
zakiahmedkgp
commented
5 years ago Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
@lukeb94 Yes, I have found a workaround. I created a new function in models.py under the class MaskRCNN() like this.
def save(self,save_path):
self.keras_model.save(save_path)and then in inspect damage model after initializing the model and loading the model weights, I called this function
model.save('model_weight.h5')
now you need to understand that this 'model' is not a keras model object but a 'class MaskRCNN()' object.
now for the .pb file, I used freeze graph which can be found here
I again saved this function inside the class MaskRCNN() and called this function after self.keras_model.fit_generator like this:
self.keras_model.fit_generator(
train_generator,
initial_epoch=self.epoch,
epochs=epochs,
steps_per_epoch=self.config.STEPS_PER_EPOCH,
callbacks=callbacks,
validation_data=val_generator,
validation_steps=self.config.VALIDATION_STEPS,
max_queue_size=100,
workers=workers,
use_multiprocessing=True,
)
#######using session and saving .pb file##
frozen_graph = self.freeze_session(K.get_session(),
output_names=[out.op.name for out in self.keras_model.outputs])
tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False)don't forget to pass 'self' in freeze_session function. this will automatically save a .pb file in your logs directory while you are training your model. you can check both, the .h5 model file and the .pb file using netron, it is a great visualizer for these architectures.
lukeb94
commented
5 years ago Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
@lukeb94 Yes, I have found a workaround. I created a new function in models.py under the class MaskRCNN() like this.
def save(self,save_path): self.keras_model.save(save_path)and then in inspect damage model after initializing the model and loading the model weights, I called this function
model.save('model_weight.h5')now you need to understand that this 'model' is not a keras model object but a 'class MaskRCNN()' object. now for the .pb file, I used freeze graph which can be found here I again saved this function inside the class MaskRCNN() and called this function after self.keras_model.fit_generator like this:self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, ) #######using session and saving .pb file## frozen_graph = self.freeze_session(K.get_session(), output_names=[out.op.name for out in self.keras_model.outputs]) tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False)don't forget to pass 'self' in freeze_session function. this will automatically save a .pb file in your logs directory while you are training your model. you can check both, the .h5 model file and the .pb file using netron, it is a great visualizer for these architectures.
Thank you for the response. I have:
added the function to model.py
called function in training proceedure after init and loading weights
Unfortunately I am still getting the 'TypeError: can't pickle_thread.RLock objects' error that I was receiving before. Have I missed something? Cheers
zakiahmedkgp
commented
5 years ago @lukeb94
Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
@lukeb94 Yes, I have found a workaround. I created a new function in models.py under the class MaskRCNN() like this.
def save(self,save_path): self.keras_model.save(save_path)and then in inspect damage model after initializing the model and loading the model weights, I called this function
model.save('model_weight.h5')now you need to understand that this 'model' is not a keras model object but a 'class MaskRCNN()' object. now for the .pb file, I used freeze graph which can be found here I again saved this function inside the class MaskRCNN() and called this function after self.keras_model.fit_generator like this:self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, ) #######using session and saving .pb file## frozen_graph = self.freeze_session(K.get_session(), output_names=[out.op.name for out in self.keras_model.outputs]) tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False)don't forget to pass 'self' in freeze_session function. this will automatically save a .pb file in your logs directory while you are training your model. you can check both, the .h5 model file and the .pb file using netron, it is a great visualizer for these architectures.
Thank you for the response. I have:
- added the function to model.py
- called function in training proceedure after init and loading weights
Unfortunately I am still getting the 'TypeError: can't pickle_thread.RLock objects' error that I was receiving before. Have I missed something? Cheers
you need to call
model.save('model_weights.h5)
in your inspect_balloon_model.ipynb, because in that notebook we first initialize the model architecture in inference mode and then load weights and then we predict the test images.
# Create model in inference mode
with tf.device(DEVICE):
model = modellib.MaskRCNN(mode="inference", model_dir=MODEL_DIR,
config=config)and
# Set path to balloon weights file
# Download file from the Releases page and set its path
# https://github.com/matterport/Mask_RCNN/releases
# weights_path = "/path/to/mask_rcnn_balloon.h5"
# Or, load the last model you trained
weights_path = model.find_last()
# Load weights
print("Loading weights ", weights_path)
model.load_weights(weights_path, by_name=True)after this you can call
model.save('model_weights.h5)
also in my previous answer if you copied the freeze_session function in your model.py under class MaskRCNN() then you would have generated the .pb file, after that you can covert your .pb file to .tflite if you want to deploy it.
lukeb94
commented
5 years ago @lukeb94
Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
@lukeb94 Yes, I have found a workaround. I created a new function in models.py under the class MaskRCNN() like this.
def save(self,save_path): self.keras_model.save(save_path)and then in inspect damage model after initializing the model and loading the model weights, I called this function
model.save('model_weight.h5')now you need to understand that this 'model' is not a keras model object but a 'class MaskRCNN()' object. now for the .pb file, I used freeze graph which can be found here I again saved this function inside the class MaskRCNN() and called this function after self.keras_model.fit_generator like this:self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, ) #######using session and saving .pb file## frozen_graph = self.freeze_session(K.get_session(), output_names=[out.op.name for out in self.keras_model.outputs]) tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False)don't forget to pass 'self' in freeze_session function. this will automatically save a .pb file in your logs directory while you are training your model. you can check both, the .h5 model file and the .pb file using netron, it is a great visualizer for these architectures.
Thank you for the response. I have:
- added the function to model.py
- called function in training proceedure after init and loading weights
Unfortunately I am still getting the 'TypeError: can't pickle_thread.RLock objects' error that I was receiving before. Have I missed something? Cheers
you need to call
model.save('model_weights.h5)in your inspect_balloon_model.ipynb, because in that notebook we first initialize the model architecture in inference mode and then load weights and then we predict the test images.# Create model in inference mode with tf.device(DEVICE): model = modellib.MaskRCNN(mode="inference", model_dir=MODEL_DIR, config=config)and
# Set path to balloon weights file # Download file from the Releases page and set its path # https://github.com/matterport/Mask_RCNN/releases # weights_path = "/path/to/mask_rcnn_balloon.h5" # Or, load the last model you trained weights_path = model.find_last() # Load weights print("Loading weights ", weights_path) model.load_weights(weights_path, by_name=True)after this you can call
model.save('model_weights.h5)also in my previous answer if you copied the freeze_session function in your model.py under class MaskRCNN() then you would have generated the .pb file, after that you can covert your .pb file to .tflite if you want to deploy it.
Hi again, thank you for another amazing response.
I have followed but when I try to load the the weights (before saving model) I get the following error. Any ideas?
I will try the freeze session option you suggested next. All I need is the .pb or .tflite file for serving. Very keen to get working!
lukeb94
commented
5 years ago @zakiahmedkgp
Thank you I successfully generated the .pb file and visualised on netron. Appreciate the help.
Is there an available method for running inference with this .pb file to test if it works? Many thanks
AI-ML-Enthusiast
commented
5 years ago Hi there, has anyone found a fix? I have successfully trained custom model with great results but outputting .h5 weights file. Need .pb model file to deploy on mobile device/cloud. Cheers
@lukeb94 Yes, I have found a workaround. I created a new function in models.py under the class MaskRCNN() like this.
def save(self,save_path): self.keras_model.save(save_path)and then in inspect damage model after initializing the model and loading the model weights, I called this function
model.save('model_weight.h5')now you need to understand that this 'model' is not a keras model object but a 'class MaskRCNN()' object. now for the .pb file, I used freeze graph which can be found here I again saved this function inside the class MaskRCNN() and called this function after self.keras_model.fit_generator like this:self.keras_model.fit_generator( train_generator, initial_epoch=self.epoch, epochs=epochs, steps_per_epoch=self.config.STEPS_PER_EPOCH, callbacks=callbacks, validation_data=val_generator, validation_steps=self.config.VALIDATION_STEPS, max_queue_size=100, workers=workers, use_multiprocessing=True, ) #######using session and saving .pb file## frozen_graph = self.freeze_session(K.get_session(), output_names=[out.op.name for out in self.keras_model.outputs]) tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False)don't forget to pass 'self' in freeze_session function. this will automatically save a .pb file in your logs directory while you are training your model. you can check both, the .h5 model file and the .pb file using netron, it is a great visualizer for these architectures.
@zakiahmedkgp Thank you for your writing. Would you give some answer please. You mean, you created two function within MaskRCNN class. These two functions are:
def save(self,save_path): self.keras_model.save(save_path)
def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True): graph = session.graph with graph.as_default(): freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or [])) output_names = output_names or [] output_names += [v.op.name for v in tf.global_variables()] input_graph_def = graph.as_graph_def() if clear_devices: for node in input_graph_def.node: node.device = "" frozen_graph = tf.graph_util.convert_variables_to_constants( session, input_graph_def, output_names, freeze_var_names) return frozen_graph
Am I right? you call the second function from the train function of the MaskRCNN class. If it is Ok, How can I write the code to create or save model in .h5 and .pb file,?
Is it in inference mode only? such as:
# GPU for training. DEVICE = "/gpu:0" with tf.device(DEVICE): model = modellib.MaskRCNN(mode="inference", config=inference_config, model_dir=MODEL_DIR) model.load_weights(weights_path, by_name=True) dataset_train = stone.StoneDataset() dataset_train.load_stone(STONE_DIR,"train") dataset_train.prepare() model.save('model_weights.h5')
My second question, What about this section:
callbacks = [ keras.callbacks.TensorBoard(log_dir=self.log_dir, histogram_freq=0, write_graph=True, write_images=False), keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=False), ]
Is it remain save_weights_only=False or true?
Please help me. I am waiting for your kind response.
zakiahmedkgp
commented
5 years ago @zakiahmedkgp
Thank you I successfully generated the .pb file and visualised on netron. Appreciate the help.
Is there an available method for running inference with this .pb file to test if it works? Many thanks
@lukeb94 you can convert this .pb file to .tflite file using tococonverter, there are answers available on StackOverflow on how to convert .pb to .tflite and then you can run your inferences using .tflite file. This might help.
Cheers!
zakiahmedkgp
commented
5 years ago @ibrahimLearning
If it is Ok, How can I write the code to create or save model in .h5 and .pb file,?
For saving the model please refer to the above comments and for .pb file you have to define a function freeze_session and then call it after the fit_generator in training.
Is it in inference mode only?
I have not tried to initialize the model in training mode during inferencing.
Is it remain save_weights_only=False or true?
it is False, it will only save weights file and not complete model.
Cheers!
AI-ML-Enthusiast
commented
5 years ago @zakiahmedkgp Thanks for your answer. But still have confusing. Please help me.
For saving the model please refer to the above comments and for .pb file you have to define a function freeze_session and then call it after the fit_generator in training.
I did already in model.py. after that I created a new file called Modelsave.py to save model, the code is here:
DEVICE = "/gpu:0" with tf.device(DEVICE): model = modellib.MaskRCNN(mode="inference", config=inference_config, model_dir=MODEL_DIR) model.load_weights(weights_path, by_name=True) model.save('model_weights.h5')
The above code is OK to save model in pb? It is run successfully but not save pb file. Because it is in inference mode, so it is not called freeze_session.
So How can I save pb file? However, Would you share your model.py please?
AI-ML-Enthusiast
commented
5 years ago @zakiahmedkgp Please, I am waiting for your response
Amir-Mehrpanah
commented
5 years ago Hi all, thank you for your support.
I'm about to convert .h5 file to .pb and I've tried both https://github.com/amir-abdi/keras_to_tensorflow and freeze_session function in @zakiahmedkgp's solution, but there are different problems with each.
The catch with the first workaround is custom_layers e.g. BatchNorm and ProposalLayer. After setting the custom_layer parameter in keras.models.load_model function this error happend:
TypeError: __init__() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'
these are input parameters of ProposalLayer's __init__ function ,to no avail, I tried to give it default values. (Giving default values will not work for PyramidROIAlign class because it's been used with various parameters)
Secondly for freezing the session I'm stuck here:
graph = session.graph inside freeze_session function
AttributeError: 'MaskRCNN' object has no attribute 'graph'
Is there any help?! thanks in advance.
Amir-Mehrpanah
commented
5 years ago Hello Maro, sad to say, I tried hard on this for a week or two in vain,🤕 despite checking various possible ways. I ran to this error when I wanted to convert H5 logs to TFLITE for android use. Eventually, I used models on the model zoo https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md that start with SSD. It might be helpful to know MASK RCNN models are computationally intensive and converted TFLITE, if possible, would be around 100mb.
On Thu, Sep 12, 2019 at 5:57 PM MaroGrigoryan notifications@github.com wrote:
@Amir-Mehrpanah https://github.com/Amir-Mehrpanah I am just wondering if you have managed to solve the problem AttributeError: 'MaskRCNN' object has no attribute 'graph'. If so, I would very much appreciate if you could give me some directions.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/matterport/Mask_RCNN/issues/1126?email_source=notifications&email_token=AK5VI6WWELNMFLLGBTKQH2LQJI7VJA5CNFSM4GEVYIL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6R37NY#issuecomment-530825143, or mute the thread https://github.com/notifications/unsubscribe-auth/AK5VI6XVP6TVEYQWLNKVZ3LQJI7VJANCNFSM4GEVYILQ .
xup5
commented
4 years ago @Amir-Mehrpanah Do you include 'proposal_count' and 'nms_threshold' when you overwrite get_config()? It may cause your h5 file doesn't contain them. And when you try to convert it, there shows a missing argument error.
Amir-Mehrpanah
commented
4 years ago @xup5 It's been a long time I'm not really sure, but I think I didn't. should I include them in the config file too?
kaanaykutkabakci
commented
4 years ago Hi all, thank you for your support. I'm about to convert .h5 file to .pb and I've tried both https://github.com/amir-abdi/keras_to_tensorflow and freeze_session function in @zakiahmedkgp's solution, but there are different problems with each. The catch with the first workaround is custom_layers e.g. BatchNorm and ProposalLayer. After setting the custom_layer parameter in keras.models.load_model function this error happend:
TypeError: __init__() missing 2 required positional arguments: 'proposal_count' and 'nms_threshold'these are input parameters of ProposalLayer's__init__function ,to no avail, I tried to give it default values. (Giving default values will not work for PyramidROIAlign class because it's been used with various parameters) Secondly for freezing the session I'm stuck here:graph = session.graph inside freeze_session functionAttributeError: 'MaskRCNN' object has no attribute 'graph'Is there any help?! thanks in advance.
Hi @Amir-Mehrpanah. Did you solve 'graph' error?
bilalahmed381
commented
4 years ago Error:
TypeError: can't pickle _thread.RLock objects
My code:
model.load_weights(WEIGHTS_FILE_TO_START_WITH, by_name=True,
exclude=["mrcnn_class_logits", "mrcnn_bbox_fc","mrcnn_bbox", "mrcnn_mask","conv0"])
model.fit_generator(
train_generator,
initial_epoch=self.epoch,
epochs=epochs,
steps_per_epoch=self.config.STEPS_PER_EPOCH,
callbacks=callbacks,
validation_data=val_generator,
validation_steps=self.config.VALIDATION_STEPS,
max_queue_size=100,
workers=workers,
use_multiprocessing=True,
)
hpeiyan
commented
4 years ago In model.py file：
keras.callbacks.ModelCheckpoint(self.checkpoint_path,
monitor='val_loss',
save_weights_only=True,
save_best_only=True),
zf921022
commented
4 years ago @zakiahmedkgp Hi，thanks for your job! I can generated the .pb file followed your way. But I still want to ask one more question, is there needed to call model.save after load weights？Since tf.train.write_graph(frozen_graph, self.log_dir, "my_model.pb", as_text=False) is called in MaskRCNN.Train function every epoch.
pedrofrodenas
commented
3 years ago I solved this problem saving the weights only on training time. After model is trained load the weighs and model architecture from code definition. After I have the model object I get self.keras_model attribute and use self.keras_model.save("Path/to/model.h5") to store model and weights
I have modified the balloon example providing one extra argument:
elif args.command == "save":
model = modellib.MaskRCNN(mode="inference", config=config,
model_dir=args.logs)
weights_path = model.find_last()
print("Loading weights ", weights_path)
model.load_weights(weights_path, by_name=True)
model.keras_model.save("full_model_keras.h5")
I want to save the network architecture of this model but getting can't pickle error. I tried some solution but not able to resolve.
_# Callbacks callbacks = [ keras.callbacks.TensorBoard(log_dir=self.log_dir, histogram_freq=0, write_graph=True, write_images=False), keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=False), ]
i tried below code as well to save the model _model.save('mask_rcnn.h5').
with open('model_architecture_subin.json', 'w') as f: f.write(model.tojson()) with yaml also i tried but same issue.
logs for can't pickle _thread.RLock objects Traceback (most recent call last): File "custom.py", line 309, in
train(model)
File "custom.py", line 220, in train
imgaug.augmenters.Affine(rotate=(-90, 90))
File "/home/hasmeet/nj/surface_crack_detection/mask_rcnn_solution/Final_Mask_RCNN_Sol/src/mrcnn/model.py", line 2453, in train
use_multiprocessing=True,
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, *kwargs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/callbacks.py", line 457, in on_epoch_end
model_json = self.model.to_json()
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/network.py", line 1239, in to_json
model_config = self._updated_config()
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/network.py", line 1187, in _updated_config
config = self.get_config()
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/network.py", line 934, in get_config
return copy.deepcopy(config) #can't pickable error
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, rv)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, rv)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, rv)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.RLock objects
logs for Not JSON Serializable
Traceback (most recent call last): File "custom.py", line 309, in
train(model)
File "custom.py", line 220, in train
imgaug.augmenters.Affine(rotate=(-90, 90))
File "/home/hasmeet/nj/surface_crack_detection/mask_rcnn_solution/Final_Mask_RCNN_Sol/src/mrcnn/model.py", line 2453, in train
use_multiprocessing=True,
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, kwargs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/training_generator.py", line 251, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/callbacks.py", line 79, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/callbacks.py", line 457, in on_epoch_end
model_json = self.model.to_json()
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/network.py", line 1240, in to_json
return json.dumps(model_config, default=get_json_type, kwargs)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/json/init.py", line 238, in dumps
**kw).encode(obj)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/home/hasmeet/.conda/envs/ai_maskrcnn/lib/python3.6/site-packages/keras/engine/network.py", line 1237, in get_json_type
raise TypeError('Not JSON Serializable:', obj)
TypeError: ('Not JSON Serializable:', <tf.Variable 'anchors/Variable:0' shape=(1, 16368, 4) dtype=float32_ref>)