search

Sense-X / UniFormer

[ICLR2022] official implementation of UniFormer
Apache License 2.0
812 stars 111 forks source link

训练视频模型时如何正确加载Imagenet的预训练权重 #102

Closed LEM0NTE closed 1 year ago

LEM0NTE commented 1 year ago

您好, 如标题, 我该如何正确加载在图像数据集的预训练权重呢? 如图中所示, 我设置了TRAIN.CHECKPOINT_INFLATE为True, 并且设定了TRAIN.CHECKPOINT_FILE_PATH为我预训练好的2D权重. 但结合目前的训练log来看似乎并没有起到预训练的效果. 所以想向您求助正确的设置方式~ image

Andy1621 commented 1 year ago

To use image pretrained model, you should set the model_path in uniformer.py: https://github.com/Sense-X/UniFormer/blob/849cd0cd3b163f84102b1a799019a689d8d3fb8a/video_classification/slowfast/models/uniformer.py#L16-L26

Then add UNIFORMER.PRETRAIN_NAME in your config.yaml or run.sh. The TRAIN.CHECKPOINT_FILE_PATH is used to load video pretrained model.

LEM0NTE commented 1 year ago

To use image pretrained model, you should set the model_path in uniformer.py:

https://github.com/Sense-X/UniFormer/blob/849cd0cd3b163f84102b1a799019a689d8d3fb8a/video_classification/slowfast/models/uniformer.py#L16-L26

Then add UNIFORMER.PRETRAIN_NAME in your config.yaml or run.sh. The TRAIN.CHECKPOINT_FILE_PATH is used to load video pretrained model.

您好, 我按照您的指导成功导入了Imagenet预训练的权重, 并成功的训练了Uniformer. 但是我修改了部分模型结构再进行训练和加载时遇到了一些无法理解的问题, 希望得到您的建议: 如图, patch embedding层的权重可以正常加载并膨胀. image 然后到第一个pos embedding层时提示错误, 但是只显示Key error, 而没有显示具体的错误原因. 我检查了checkpoint['model']['blocks1.0.pos_embed.weight']的shape与Uniformer是一致的, 也有具体的权重值. image 希望能得到您的一些建议, 十分感谢!

Andy1621 commented 1 year ago

key error的意思是你的模型缺少'blocks1.0.pos_embed.weight'，你可以检查一下修改模型结构的时候是不是删除了block1的pos_embed。