coatnet, bias weights per head

leondgarse / keras_cv_attention_models

Keras beit,caformer,CMT,CoAtNet,convnext,davit,dino,efficientdet,edgenext,efficientformer,efficientnet,eva,fasternet,fastervit,fastvit,flexivit,gcvit,ghostnet,gpvit,hornet,hiera,iformer,inceptionnext,lcnet,levit,maxvit,mobilevit,moganet,nat,nfnets,pvt,swin,tinynet,tinyvit,uniformer,volo,vanillanet,yolor,yolov7,yolov8,yolox,gpt2,llama2, alias kecam

MIT License

595 stars 95 forks source link

coatnet, bias weights per head #10

Closed masterkni6 closed 2 years ago

masterkni6 commented 2 years ago

Your coatnet seems to only create 1 bias weight set when the paper says 1 per head Screenshot_2021-10-29_18-25-19

leondgarse commented 2 years ago

OK. I've updated it using MultiHeadRelativePositionalEmbedding from beit. But seems there are still something not matching, as my parameter counts not meeting paper reports...

Model	self defined	paper reported
CoAtNet0	23.86M	25M
CoAtNet1	41.05M	42M
CoAtNet2	72.67M	75M
CoAtNet3	162.46M	168M
CoAtNet4	270.90M	275M