search

ucbdrive / few-shot-object-detection

Implementations of few-shot object detection benchmarks
Apache License 2.0
1.08k stars 225 forks source link

fine-tune result is great but demo can't detected anything #135

Open AISoltani opened 2 years ago

AISoltani commented 2 years ago

Hi I used a custom dataset on this project, which has great result in the fine tune step( eval result was great ) , but when I try to use the demo to detect the image, it does not detect anything, what is my problem?....base model, combine model and novel model can detect but final model from fine tune step can't!!!...(my dataset type was like pascal voc, jpg with xml).

i tested with both way like base train >> random initial >> fine tune and also base train >> random initial>> novel train >> combine and fine tune...all of them have good result in terminal but final model can't detected anything with demo ...this is my final result from fine tune stage:

result

thanks for help.

thomasehuang commented 2 years ago

What command are you running when using the demo? Seems like an issue on the demo side rather than the model.

AISoltani commented 2 years ago

What command are you running when using the demo? Seems like an issue on the demo side rather than the model.

First thanks for your answer, I tried all week and could not solve the problem and pls support us until the complete solution pls.... well ...i used this command:

python3 -m demo.demo --config-file configs/PascalVOC-detection/split1/faster_rcnn_R_101_FPN_ft_all1_1shot.yaml --input image1.jpg --opts MODEL.WEIGHTS checkpoints/voc/faster_rcnn/faster_rcnn_R_101_FPN_ft_normalized_all1_1shot_randnovel/model_final.pth

this model_final was generated from fine tuning stage and my config is faster_rcnn_R_101_FPN_ft_all1_1shot.yaml and inside the config:

_BASE_: "../../Base-RCNN-FPN.yaml"
MODEL:
  WEIGHTS: "checkpoints/voc/faster_rcnn/faster_rcnn_R_101_FPN_all2/model_reset_combine.pth"
  MASK_ON: False
  RESNETS:
    DEPTH: 101
  ROI_HEADS:
    NUM_CLASSES: 32
    OUTPUT_LAYER: "CosineSimOutputLayers"
    FREEZE_FEAT: True
  BACKBONE:
    FREEZE: True
  PROPOSAL_GENERATOR:
    FREEZE: True
INPUT:
  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
  MIN_SIZE_TEST: 800
DATASETS:
  TRAIN: ('voc_2007_trainval_all1_1shot',)
  TEST: ('voc_2007_test_all1',)
SOLVER:
  IMS_PER_BATCH: 2
  BASE_LR: 0.001
  STEPS: (3000,)
  MAX_ITER: 4000
  CHECKPOINT_PERIOD: 500
  WARMUP_ITERS: 0
OUTPUT_DIR: "checkpoints/voc/faster_rcnn/faster_rcnn_R_101_FPN_ft_normalized_all1_1shot_randnovel"

i have 32 class if i use novel train i need to choose my combine model for weights in fine tuning and if i use random intial i choose model surgery for weights but at all demo can not detected after fine tuning. I will rewrite the way I do my work in detail from the beginning. Maybe you can better guide me. first my dataset is like pascal voc, jpg for images and xml for labels. i use this command for base training :

CUDA_VISIBLE_DEVICES=0 python3 -m tools.train_net --config-file configs/PascalVOC-detection/split1/faster_rcnn_R_101_FPN_base1.yaml

and inside the base config is:

_BASE_: "../../Base-RCNN-FPN.yaml"
MODEL:
  WEIGHTS: "/home/cudaa/Downloads/R-101.pkl"
  MASK_ON: False
  RESNETS:
    DEPTH: 101
  ROI_HEADS:
    NUM_CLASSES: 24
INPUT:
  MIN_SIZE_TRAIN: (480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800)
  MIN_SIZE_TEST: 800
DATASETS:
  TRAIN: ('voc_2007_trainval_base1',)
  TEST: ('voc_2007_test_base1',)
SOLVER:
  STEPS: (12000, 16000)
  MAX_ITER: 18000  # 17.4 epochs
  WARMUP_ITERS: 100
OUTPUT_DIR: "checkpoints/voc/faster_rcnn/faster_rcnn_R_101_FPN_base-final"

and because i have 32 class , 24 class for base and 8 class for novel... after random initialize step i use this config faster_rcnn_R_101_FPN_ft_all1_1shot.yaml for fine tuning on surgery model (or use this command with 10shot) as i said above, and use this also for demo. Why my result in terminal was good but my model after fine tuning stage could not detected anything with demo?!

thomasehuang commented 2 years ago

Thanks for the detailed explanation. I'm actually asking for the command you used when running the demo.py. My suspicion of the issue is the demo, since your evaluation results are good. Usually, not being able to detect anything means the model weights aren't loaded correctly.

AISoltani commented 2 years ago

Thanks for the detailed explanation. I'm actually asking for the command you used when running the demo.py. My suspicion of the issue is the demo, since your evaluation results are good. Usually, not being able to detect anything means the model weights aren't loaded correctly.

What is your suggestion to solve this problem?

thomasehuang commented 2 years ago

Oh, sorry I didn't see you already added the command you ran for the demo. It looks good to me, so I'll need to reproduce your issue on my side. Can you send me information about your environment? Python and packages versions that you used to run the code.

AISoltani commented 2 years ago

Oh, sorry I didn't see you already added the command you ran for the demo. It looks good to me, so I'll need to reproduce your issue on my side. Can you send me information about your environment? Python and packages versions that you used to run the code.

Sure, here you are:

 Python 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59) 
 [GCC 7.5.0] on linux
 absl-py==0.13.0
 cachetools==4.2.2
 certifi==2021.5.30
 charset-normalizer==2.0.4
 cloudpickle==1.6.0
 cycler==0.10.0
 Cython==0.29.24
 dataclasses==0.8
 detectron2==0.2.1+cu102
 future==0.18.2
 fvcore==0.1.5.post20210812
 google-auth==1.34.0
 google-auth-oauthlib==0.4.5
 grpcio==1.39.0
 idna==3.2
 imagesize==1.2.0
 importlib-metadata==4.6.4
 iopath==0.1.9
 kiwisolver==1.3.1
 lvis==0.5.3
 Markdown==3.3.4
 matplotlib==3.3.4
 mkl-fft==1.3.0
 mkl-random==1.1.1
 mkl-service==2.3.0
 mock==4.0.3
 numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603487797006/work
 oauthlib==3.1.1
 olefile==0.46
 opencv-python==4.5.3.56
 Pillow @ file:///tmp/build/80754af9/pillow_1625649052827/work
 portalocker==2.3.0
 protobuf==3.17.3
 pyasn1==0.4.8
 pyasn1-modules==0.2.8
 pycocotools==2.0.2
 pydot==1.4.2
 pyparsing==2.4.7
 python-dateutil==2.8.2
 PyYAML==5.4.1
 requests==2.26.0
 requests-oauthlib==1.3.0
 rsa==4.7.2
 six @ file:///tmp/build/80754af9/six_1623709665295/work
 tabulate==0.8.9
 tensorboard==2.6.0
 tensorboard-data-server==0.6.1
 tensorboard-plugin-wit==1.8.0
 termcolor==1.1.0
 torch==1.6.0
 torchvision==0.7.0
 tqdm==4.62.0
 typing-extensions==3.10.0.0
 urllib3==1.26.6
 Werkzeug==2.0.1
 yacs==0.1.8
 zipp==3.5.0

I dont know what happening after fine tuning!!!...base model result in terminal is good and also demo can detected with base model but after fine tuning this is very strange that result in terminal is perfect but demo could not detected any instances!!!...at all if you need any other thing like my model or any ....i can upload everything for you because i worked a lot of time and need to be successful...really thanks for your help.

this is also my final log from base and final step (fine tune stage): log-base train.txt log-fine tune train.txt :

AISoltani commented 2 years ago

I'm still waiting for you Thomas, miss you much, just please come back with good news :)...I tried a lot but I still did not get any results !

thomasehuang commented 2 years ago

Apologies for the delay. Given that the demo works on the base model, that means the demo and model inference has no issues. That means the model itself has some issues. Can you provide some more details?

AISoltani commented 2 years ago

Apologies for the delay. Given that the demo works on the base model, that means the demo and model inference has no issues. That means the model itself has some issues. Can you provide some more details?

  • What images are you running the demo on? What classes do they contain (base or novel)? Have you tried many images and none of them work?
  • Have you tried running the demo on images from the training/validation set? Your fine-tuned model should be able to detect objects in the training images at least.
  • Can you provide the terminal output after you run the demo with the fine-tuned model?

Thanks for answering, I tested train image,val image and also test image on the demo and nothing detected (novel and base not detected)...0 instances!!!...i tried many image and none of them work. I know my fine-tuned model should be able to detect objects in the training images at least but nothing detected!!!

this is the demo command and the terminal output after running demo :

demo command :

cudaa@cudaa-All-Series:~/Desktop/fsod$ python3 -m demo.demo --config-file configs/PascalVOC-detection/split1/faster_rcnn_R_101_FPN_ft_all1_5shot.yaml --input 000091.jpg --opts MODEL.WEIGHTS lastmodel/model_final.pth terminal output:

[10/27 15:18:04 detectron2]: Arguments: Namespace(confidence_threshold=0.5, config_file='configs/PascalVOC-detection/split1/faster_rcnn_R_101_FPN_ft_all1_5shot.yaml', input=['000091.jpg'], opts=['MODEL.WEIGHTS', 'lastmodel/model_final.pth'], output=None, video_input=None, webcam=False)
froze backbone parameters
froze proposal generator parameters
froze roi_box_head parameters
[10/27 15:18:07 fvcore.common.checkpoint]: [Checkpointer] Loading from lastmodel/model_final.pth ...
/home/cudaa/Desktop/fsod/fsdet/modeling/roi_heads/fast_rcnn.py:115: UserWarning: This overload of nonzero is deprecated:
    nonzero()
Consider using one of the following signatures instead:
    nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629427286/work/torch/csrc/utils/python_arg_parser.cpp:766.)
  filter_inds = filter_mask.nonzero()
[10/27 15:18:07 detectron2]: 000091.jpg: detected 0 instances in 0.44s

What should i do Thomas?

ElifErarslan1 commented 2 years ago

the same problem also occurs in my case. did you solve it?

AISoltani commented 2 years ago

well done...Thomas escaped, runaway!!!

WumingzheAI commented 2 years ago

hahaha

DanLiu0623 commented 2 years ago

did you solve it? @AISoltani @WumingzheAI

WumingzheAI commented 2 years ago

what is the problem

------------------ 原始邮件 ------------------ 发件人: "ucbdrive/few-shot-object-detection" @.>; 发送时间: 2022年2月9日(星期三) 下午2:33 @.>; @.**@.>; 主题: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135)

did you solve it? @AISoltani @WumingzheAI

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

DanLiu0623 commented 2 years ago

what is the problem … ------------------ 原始邮件 ------------------ 发件人: "ucbdrive/few-shot-object-detection" @.>; 发送时间: 2022年2月9日(星期三) 下午2:33 @.>; @.**@.>; 主题: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) did you solve it? @AISoltani @WumingzheAI — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

sorry,I @ wrong person, the problem same as fine-tune result is great but demo can't detected anything

WumingzheAI commented 2 years ago

你用的什么权重

---Original--- From: "Liu @.> Date: Thu, Feb 10, 2022 08:59 AM To: @.>; Cc: @.**@.>; Subject: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135)

what is the problem … ------------------ 原始邮件 ------------------ 发件人: "ucbdrive/few-shot-object-detection" @.>; 发送时间: 2022年2月9日(星期三) 下午2:33 @.>; @.@.>; 主题: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) did you solve it? @AISoltani @WumingzheAI — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

sorry,I @ wrong person, the problem same as fine-tune result is great but demo can't detected anything

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.***>

DanLiu0623 commented 2 years ago

@AISoltani I need your help,I want to know did you solve the problem.I can detect base object after base training,but can not detect everything after finetune.

DanLiu0623 commented 2 years ago

你用的什么权重 … ---Original--- From: "Liu @.> Date: Thu, Feb 10, 2022 08:59 AM To: @.>; Cc: @.**@.>; Subject: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) what is the problem … ------------------ 原始邮件 ------------------ 发件人: "ucbdrive/few-shot-object-detection" @.>; 发送时间: 2022年2月9日(星期三) 下午2:33 @.>; @.@.>; 主题: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) did you solve it? @AISoltani @WumingzheAI — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.> sorry,I @ wrong person, the problem same as fine-tune result is great but demo can't detected anything — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.>

我是base9类，novel3类，base类训练完的模型可以正常检测base9类，fine tune完之后的模型连9类都检测不到，所有的12类也检测不到。但是训练log里面eval的结果还可以。

ysw774977 commented 2 years ago

你用的什么权重 … ---Original--- From: "Liu @.**> Date: Thu, Feb 10, 2022 08:59 AM To: @.**>; Cc: @.**@.**>; Subject: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) what is the problem … ------------------ 原始邮件 ------------------ 发件人: "ucbdrive/few-shot-object-detection" @.>; 发送时间: 2022年2月9日(星期三) 下午2:33 @.>; @.@.>; 主题: Re: [ucbdrive/few-shot-object-detection] fine-tune result is great but demo can't detected anything (#135) did you solve it? @AISoltani @WumingzheAI — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.**> sorry,I @ wrong person, the problem same as fine-tune result is great but demo can't detected anything — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.Message ID: @.**>

我是base9类，novel3类，base类训练完的模型可以正常检测base9类，fine tune完之后的模型连9类都检测不到，所有的12类也检测不到。但是训练log里面eval的结果还可以。

请问问题解决了吗 我也是只能检测到base类，novel类都检测不到