Closed afiaka87 closed 3 years ago
@afiaka87 nice, glad to hear it is working! :)
@kobiso @lucidrains @janEbert can any of you speak as to why the axial and conv-like attention seem to require so much more memory than using just 'full' on its own?
my understanding was that these layers operated on more efficient feature maps. but i may need to revisit the topic.
ohh I understand why, I'll get it fixed by Friday!
ohh I understand why, I'll get it fixed by Friday!
does that mean I should wait for training a new model (i.e. breaking changes?) or is it safe to do so :)
@TheodoreGalanos it won't be breaking! It'll simply be more memory efficient attention , train away :)
Great work @lucidrains
# Original post by @kobiso https://github.com/lucidrains/DALLE-pytorch/discussions/131#discussioncomment-640446
Go give them a rocket emoji!
Attention type ('full', 'axial_row', 'axial_col', 'conv_like') works
Experimental setting
Computational cost
Training log
Results
Originally posted by @kobiso in https://github.com/lucidrains/DALLE-pytorch/discussions/131#discussioncomment-640446