search

hkchengrex / Tracking-Anything-with-DEVA

[ICCV 2023] Tracking Anything with Decoupled Video Segmentation
https://hkchengrex.com/Tracking-Anything-with-DEVA/
Other
1.27k stars 129 forks source link

Fail to tracking anything in green plant scenes. Any suggestion? #107

Closed garrisonz closed 3 months ago

garrisonz commented 3 months ago

DEVA fail to automatic tracking anything with flower scene in LLFF dataset. The images in the Annotations folder are all black. Any suggestion? Thanks!

The output is following

Configuration: {'model': './saves/DEVA-propagation.pth', 'output': '/home/zhangyupeng/w/3drecon/LangSplat/dataset/nerf_llff_data/flower/mask/video_mask_auto/', 'save_all': False, 'amp': True, 'key_dim': 64, 'value_dim': 512, 'pix_feat_dim': 512, 'disable_long_term': False, 'max_mid_term_frames': 10, 'min_mid_term_frames': 5, 'max_long_term_elements': 10000, 'num_prototypes': 128, 'top_k': 30, 'mem_every': 5, 'chunk_size': 4, 'size': 480, 'GROUNDING_DINO_CONFIG_PATH': './saves/GroundingDINO_SwinT_OGC.py', 'GROUNDING_DINO_CHECKPOINT_PATH': './saves/groundingdino_swint_ogc.pth', 'DINO_THRESHOLD': 0.35, 'DINO_NMS_THRESHOLD': 0.8, 'SAM_ENCODER_VERSION': 'vit_h', 'SAM_CHECKPOINT_PATH': './saves/sam_vit_h_4b8939.pth', 'MOBILE_SAM_CHECKPOINT_PATH': './saves/mobile_sam.pt', 'SAM_NUM_POINTS_PER_SIDE': 24, 'SAM_NUM_POINTS_PER_BATCH': 64, 'SAM_PRED_IOU_THRESHOLD': 0.88, 'SAM_OVERLAP_THRESHOLD': 0.8, 'img_path': '/home/zhangyupeng/w/3drecon/LangSplat/dataset/nerf_llff_data/flower/images', 'detection_every': 5, 'num_voting_frames': 3, 'temporal_setting': 'semionline', 'max_missed_detection_count': 5, 'max_num_objects': 100, 'sam_variant': 'original', 'suppress_small_objects': False, 'enable_long_term': True, 'enable_long_term_count_usage': False}
in demo......
  6%|█████████▏                                                                                                                                                   | 2/34 [00:03<00:44,  1.41s/it]
Restricted license - for non-production use only - expires 2025-11-24
GurobiError, falling back to pulp
UserWarning: Number of objects exceeded maximum (--max_num_objects 100); discarding new objects
RuntimeWarning: Empty object mask!
RuntimeWarning: Trying to segment without any memory!
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34/34 [00:37<00:00,  1.10s/it]

flower scene image is show in below video.

https://github.com/user-attachments/assets/ce932ac7-ea80-4d07-ad85-7dc20a65e569

garrisonz commented 3 months ago

Actually, DEVA also fail to automaticly tracking anything in leaves, orchids scene in the LLFF dataset. All the failed scene is green plant. On the other hand, building scenes are work fine with DEVA tracking.

hkchengrex commented 3 months ago

Are you using the automatic mode or the text-prompt mode? The first thing to check would be whether the corresponding image model produces anything useful.

garrisonz commented 3 months ago

@hkchengrex DEVA fails to track using automatic mode. However, DEVA tracks flower successfully with text prompt "flower".

It seems a bug using automatical mode.

garrisonz commented 3 months ago

Text prompt mode meet my need. So, I will close this issue. thanks for response！