search

Visual-Behavior / detr-tensorflow

Tensorflow implementation of DETR : Object Detection with Transformers
MIT License
168 stars 53 forks source link

Model saving issue #24

Closed myopengit closed 3 years ago

myopengit commented 3 years ago

I have trained the model for around 230 epoch on the coco dataset using your "train_coco.py". I saved the model using detr.save_weights('weights.h5'). When testing on the same image, I got different results when I loading back the saved weights... (and the performance is not good at all) It seems this is an issue with keras （https://github.com/keras-team/keras/issues/4875#). Can you provide a solution on how to save your model? Thanks.

PhanTask commented 3 years ago

I use save_weights and load_weights to save and load the trained weights and everything works fine for me. I finetuned the model on my own dataset. Did you normalize the input coco images before doing inference? Also you may check if you set training=False for detr when doing inference.

simpad2409 commented 3 years ago

Hello, @myopengit . Can I ask you how did you solve it? I think I have the same problem as you. I save and restore the checkpoints with save_weights and load_weights to save and load the trained weights. The checkpoints I retrieve for inference do not give me the same result.

Thanks in advance.

simpad2409 commented 3 years ago

@thibo73800 checkpoints don't save real weights. How can I solve this problem? I can't save my model ... Help me. thank you.

yuh8 commented 3 years ago

@simpad2409 You probably used custom functions/schedules/layers which cannot be serialized by keras. I have the exact same issue; I cannot use save model, and saving and loading weights give very poor results. The way I got away with this is to used to the following saving procedure. tf.keras.models.save_model(model, './generator_model/', include_optimizer=False) model_new = tf.keras.models.load_model('./generator_model/', compile=False) model_new.compile(optimizer=get_optimizer(), loss=loss_func_gen) res = model_new.evaluate(data_iterator_test('generator_data/test_data/df_test.csv'), return_dict=True) Pay attention to "include_optimizer=False". I have used a custom learning rate schedule which is part of the optimizer and cannot be saved using model.save(). By setting this config to false, I am able to save model. In the load_model, I also set the compile to False. This will load the model without optimizer and without compilation. Subsequently, I compiled again and results are the same.

P.S. I am using tensorflow 2.6.0 with keras 2.6.0 on python 3.8.3

simpad2409 commented 3 years ago

@yuh8 Thanks a lot for the answer! You have been very kind ... I'll try as soon as possible and I'll let you know.

I'm trying to finetun DETR for people detection, feeding the Crowd Human dataset. The results I get, however, are no better than the original model, albeit verticalized to pedestrian detection. Do you have any advice for me?

I'm using the same code as the "finetune_hardhat.py" file, with batchsize to 32.

Thanks. :)