Closed TinnyFlames closed 2 years ago
@TinnyFlames @TinnyFlames Not straightforward, we have tried to pre-train the vit-base 100 epochs with hog prediction in ImageNet-1k under the MAE architecture. The hog targets is slightly higher than the pixel norm (82.7% vs 82.4%).
Hi @mx-mark! Thanks for your reply. Is it possible to share suggestions on implementing the mask code? I tried to implement maskfeat for Imagenet but was confused about the mask part. The image size is (224,224,3) but the hog feature size is (14,14,108). If we randomly mask image patches, how to mask the hog features correctly since the dimension is not matched?
CubeMaskGenerator
is a little bit obscure to rewrite for me.
@mx-mark May I ask you how you calculate your loss function in the imagenet? just as the same as this? https://github.com/mx-mark/VideoTransformer-pytorch/blob/main/video_transformer.py#L899
@RechelTeamo right, the loss minimizes the L2 distance between the predicted and original HOG feature.
Thanks for your answer. I try this loss function. But I meet the problem that my loss becomes NaN. I find this problem in the blocks(x). I think the problem is lr. May I ask about your lr setting? I use blr=1e-4 and batch size 1024 in 8 V100 GPU.
@RechelTeamo the setting for what, pre-training or fine-tuning?
for pre-training
@RechelTeamo There are some related problems reported in the original MAE repos. You can check if it works https://github.com/facebookresearch/mae/issues/65, https://github.com/facebookresearch/mae/issues/42
@RechelTeamo For my pretraining settings, the blr is 1.5e-4 and the effective batch size is 4096.
@mx-mark Oh thanks a lot. I set blr: 5.00e-05 and batch 1024 (actual lr=2e-4)now it's epoch 16. I will report my result when it is finished.
My blr=1e-4 batch=1024 (actual lr=4e-4) will loss NaN.
Hi! Thanks for your code. For maskfeat code, is it possible to use it on imagenet dataset to reproduce the paper's result?