Issue in using the generated Fine-tuned weight file

[tarunsk1998]

I followed the document to set the library code. I generated my custom dataset as provided in the sample CSV file.

I used the groundingdino_swint_ogc.pth pre-trained model for the finetuning purpose and its corresponding config file. I ran the train.py provided in this code for training and the new weights were saved in my machine.

However, the issue appeared when I tried to test the model using the test.py file provided in this code. I configured the model path to the generated fine-tuned model while trying to predict a test image.

The Error is as below: File "Grounding-Dino-FineTuning\groundingdino\util\inference.py" in Line number 37: model.load_state_dict(clean_state_dict(checkpoint["model"]), strict=False) KeyError: 'model'

Let me know how this issue can be rectified. Could the train.py script be missing something?

[Asad-Ismail]

Hi @tarunsk1998 thank you for pointing it out should be fixed now see this commit https://github.com/Asad-Ismail/Grounding-Dino-FineTuning/commit/757928cf0c94a55b6290bf92165d84453cf2363f#r138678714

[tarunsk1998]

Hi @Asad-Ismail,

I tried the latest code, but faced a follow up issue while running the test.py file.

The below is the error, C:\venv\lib\site-packages\transformers\modeling_utils.py:993: FutureWarning: The device argument is deprecated and will be removed in v5 of Transformers. warnings.warn( Traceback (most recent call last): File "C:\Grounding-Dino-FineTuning\test.py", line 124, in process_video(model_weights=model_weights, model_config=model_config, File "C:\Grounding-Dino-FineTuning\test.py", line 90, in process_video boxes, logits, phrases = predict( File "C:\Grounding-Dino-FineTuning\groundingdino\util\inference.py", line 76, in predict outputs = model(image[None], captions=[caption]) File "C:\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "C:\Grounding-Dino-FineTuning\groundingdino\models\GroundingDINO\groundingdino.py", line 303, in forward self.set_image_tensor(samples) File "C:\Grounding-Dino-FineTuning\groundingdino\models\GroundingDINO\groundingdino.py", line 212, in set_image_tensor self.features, self.poss = self.backbone(samples) File "C:\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\Grounding-Dino-FineTuning\groundingdino\models\GroundingDINO\backbone\backbone.py", line 151, in forward xs = self0 File "C:\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "C:\Grounding-Dino-FineTuning\groundingdino\models\GroundingDINO\backbone\swin_transformer.py", line 716, in forward x = self.patch_embed(x) File "C:\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "C:\Grounding-Dino-FineTuning\groundingdino\models\GroundingDINO\backbone\swin_transformer.py", line 491, in forward x = self.proj(x) # B C Wh Ww File "C:\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "C:\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "C:\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: Input type (unsigned char) and bias type (float) should be the same

Let me know if this is a known issue and how I can rectified this issue.