search

open-mmlab / mmdetection

OpenMMLab Detection Toolbox and Benchmark
https://mmdetection.readthedocs.io
Apache License 2.0
28.55k stars 9.29k forks source link

grounding dino swin large finetune error #11570

Open JeremyLin886 opened 3 months ago

JeremyLin886 commented 3 months ago

I have prepared my own data set according to the steps of 'Example of Fine-tuning Custom Dataset' in the https://github.com/open-mmlab/mmdetection/blob/main/configs/mm_grounding_dino/usage_zh-CN.md#%E8%87%AA%E5%AE%9A%E4%B9%89%E6%95%B0%E6%8D%AE%E9%9B%86%E5%BE%AE%E8%B0%83%E8%AE%AD%E7%BB%83%E6%A1%88%E4%BE%8B, and the format is the same as that of 'cat' data set provided by the author. When I use grounding_dino_swin-l for finetune, pretrain weight is "grounding_dino_swin-l_pretrain_obj365_goldg-34dcdc53.pth" , I found that when I set frozen_stages=-1 in the model, the code training works fine, but if I set frozen_stages to another value, such as 1 or 2, I get an error: Traceback (most recent call last): 2024-03-19 21:23 File "/lpai/volumes/cloudmodel-muses/aaa_lin/mmdetection/./tools/train.py", line 121, in <module> 2024-03-19 21:23 main() 2024-03-19 21:23 File "/lpai/volumes/cloudmodel-muses/aaa_lin/mmdetection/./tools/train.py", line 117, in main 2024-03-19 21:23 runner.train() 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/runner.py", line 1777, in train 2024-03-19 21:23 model = self.train_loop.run() # type: ignore 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 96, in run 2024-03-19 21:23 self.run_epoch() 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 112, in run_epoch 2024-03-19 21:23 self.run_iter(idx, data_batch) 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/runner/loops.py", line 128, in run_iter 2024-03-19 21:23 outputs = self.runner.model.train_step( 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/model/wrappers/distributed.py", line 121, in train_step 2024-03-19 21:23 losses = self._run_forward(data, mode='loss') 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/mmengine/model/wrappers/distributed.py", line 161, in _run_forward 2024-03-19 21:23 results = self(**data, mode=mode) 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1510, in _wrapped_call_impl 2024-03-19 21:23 return self._call_impl(*args, **kwargs) 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1519, in _call_impl 2024-03-19 21:23 return forward_call(*args, **kwargs) 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1505, in forward 2024-03-19 21:23 inputs, kwargs = self._pre_forward(*inputs, **kwargs) 2024-03-19 21:23 File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1399, in _pre_forward 2024-03-19 21:23 if torch.is_grad_enabled() and self.reducer._rebuild_buckets(): 2024-03-19 21:23 RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argumentfind_unused_parameters=Truetotorch.nn.parallel.DistributedDataParallel, and by 2024-03-19 21:23 making sure allforwardfunction outputs participate in calculating loss. 2024-03-19 21:23 If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module'sforwardfunction. Please include the loss function and the structure of the return value offorwardof your module when reporting this issue (e.g. list, dict, iterable). 2024-03-19 21:23 Parameter indices which did not receive grad for rank 1: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 326 327 2024-03-19 21:23 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

Here is my configs, I use 8 card and set batch_size=4 for training_dataloader, batch_size=1 for val_dataloader: `base = 'grounding_dino_swin-t_pretrain_obj365.py'

data_root = 'some_root_path/' class_name = ("signal triangle", "horizontal tire", "cardboard box", ) num_classes = len(class_name) metainfo = dict(classes=class_name, palette=[(220, 20, 60)])

num_levels = 5 model = dict( use_autocast=True, num_feature_levels=num_levels, backbone=dict( delete=True, type='SwinTransformer', pretrain_img_size=384, embed_dims=192, depths=[2, 2, 18, 2], num_heads=[6, 12, 24, 48], window_size=12, mlp_ratio=4, qkv_bias=True, qk_scale=None, drop_rate=0., attn_drop_rate=0., drop_path_rate=0.2, patch_norm=True, out_indices=(0, 1, 2, 3),

Please only add indices that would be used

    # in FPN, otherwise some parameter will not be used
    with_cp=True,
    convert_weights=True,
    frozen_stages=1,
    init_cfg=None),
neck=dict(in_channels=[192, 384, 768, 1536], num_outs=num_levels),
encoder=dict(layer_cfg=dict(self_attn_cfg=dict(num_levels=num_levels))),
decoder=dict(layer_cfg=dict(cross_attn_cfg=dict(num_levels=num_levels))))

train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', with_bbox=True), dict(type='RandomFlip', prob=0.5), dict(type='RandomChoice', transforms=[ [ dict( type='RandomChoiceResize', scales=[(480, 1333), (512, 1333), (544, 1333), (576, 1333), (608, 1333), (640, 1333), (672, 1333), (704, 1333), (736, 1333), (768, 1333), (800, 1333)], keep_ratio=True) ] ]), dict(type='PackDetInputs', meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'scale_factor', 'flip', 'flip_direction', 'text', 'custom_entities')) ]

train_dataloader = dict( dataset=dict( delete=True, type='CocoDataset', data_root=data_root, metainfo=metainfo, return_classes=True, pipeline=train_pipeline, filter_cfg=dict(filter_empty_gt=False, min_size=32), ann_file='trainod.json', data_prefix=dict(img='images')))

val_dataloader = dict( dataset=dict( metainfo=metainfo, data_root=data_root, ann_file='val_od.json', data_prefix=dict(img='images')))

test_dataloader = val_dataloader

val_evaluator = dict(ann_file=data_root + 'test_od.json',) test_evaluator = val_evaluator

max_epoch = 20

default_hooks = dict( checkpoint=dict(interval=1, max_keep_ckpts=1, save_best='auto'), logger=dict(type='LoggerHook', interval=5)) train_cfg = dict(max_epochs=max_epoch, val_interval=1)

param_scheduler = [ dict( type='MultiStepLR', begin=0, end=max_epoch, by_epoch=True, milestones=[15], gamma=0.1) ]

optim_wrapper = dict( optimizer=dict(lr=0.0001), paramwise_cfg=dict( custom_keys={ 'absolute_pos_embed': dict(decay_mult=0.), 'backbone': dict(lr_mult=0.0), 'language_model': dict(lr_mult=0.0) }))

load_from = '/model_path/grounding_dino_swin-l_pretrain_obj365_goldg-34dcdc53.pth' # noqa ` Thank you for your help

taltlusty commented 3 months ago

following ...

sharat29ag commented 3 months ago

hi @JeremyLin886 , for finetuning Grounding DINO, how much GPU memory is required? Is it same as of training?

JeremyLin886 commented 3 months ago

hi @JeremyLin886 , for finetuning Grounding DINO, how much GPU memory is required? Is it same as of training?

hi, for swin-t, i use 4 A10 is ok, for swin-large, A10 is not work, i use 8 A800, i think finetuning is same as training since i set frozen_stage=-1

sharat29ag commented 3 months ago

hi @JeremyLin886 , for finetuning Grounding DINO, how much GPU memory is required? Is it same as of training?

hi, for swin-t, i use 4 A10 is ok, for swin-large, A10 is not work, i use 8 A800, i think finetuning is same as training since i set frozen_stage=-1

Thanks for the quick reply, also for a custom dataset what about the textual description? Just the image category can be used?

ShaohonChen commented 1 month ago

maybe same with https://github.com/open-mmlab/mmdetection/issues/8208