search

Caiyun-AI / DCFormer

MIT License
186 stars 15 forks source link

How to embed DCMHA into ViT #12

Open huangliqwe2020 opened 3 months ago

huangliqwe2020 commented 3 months ago

Dear author, after reading your paper, I have gained a lot from it. I hope to apply DCMHA to visual models. I noticed that your paper's experimental part embedded DCMHA into ViT for verification, but the provided code does not seem to include this part. If you could open source this part of the code, we would greatly appreciate it!

Lisennlp commented 3 months ago

We implemented our work based on google-research/big_vision. You can find the specific instructions at README in Lisennlp/big_vision repository. There have some Python configuration files ending with 'dc', which are the configurations for different model sizes used in our paper’s ViT experiments. The code of DCMHA mainly occurs in the MultiHeadDotProductAttention class within the vit.py model file.