How to add support for more custom models in pre-training

keyu-tian / SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

https://arxiv.org/abs/2301.03580

MIT License

1.42k stars 82 forks source link

How to add support for more custom models in pre-training #17

Closed zero0kiriyu closed 1 year ago

zero0kiriyu commented 1 year ago

This project is very cool, but it would be nice if you provided a simple tutorial to support more model architectures :)

keyu-tian commented 1 year ago

@zero0kiriyu Thank you for the advice. We'll get to work on this. For now u could check https://github.com/keyu-tian/SparK/tree/main/pretrain#some-details-how-we-mask-images-and-how-to-set-the-patch-size.

We're gonna refactor the code (make it more easier to use a custom model) and add a tutorial maybe by next Tuesday. Would comment here again when we finish this.

zero0kiriyu commented 1 year ago

It would be great! Also, when I tried to modify the code, I noticed that the number of channels in each layer must be reduced by half, but not all models follow this rule. For example, some lightweight networks used in my projects have channel numbers like 24, 48, 96, 128. Perhaps you can provide more flexibility to support more custom models.

keyu-tian commented 1 year ago

@zero0kiriyu You can kindly read https://github.com/keyu-tian/SparK/tree/main/pretrain#tutorial-for-customizing-your-own-cnn-model for how to customize your cnn model, hope you enjoy it!