search

SHI-Labs / Neighborhood-Attention-Transformer

Neighborhood Attention Transformer, arxiv 2022 / CVPR 2023. Dilated Neighborhood Attention Transformer, arxiv 2022
MIT License
1.04k stars 85 forks source link

About the detail of NATTENQKRPBFunction and NATTENAVFunction #7

Closed myt889 closed 2 years ago

myt889 commented 2 years ago

Hi, first congratulations on your great work! I am a newer to Transformer. I have successfully installed the NA version of cuda. But when I read your core code about NA, I can not understand the detailed codes of this two functions:NATTENQKRPBFunction and NATTENAVFunction. Could you explian ? Thank you so much.

alihassanijr commented 2 years ago

Hello and thank you for your interest.

Those are Autograd functions, which call the forward and backward pass CUDA kernels. If you like to dig deeper on those, you'd have to refer to the CUDA kernel.

myt889 commented 2 years ago

Hello and thank you for your interest.

Those are Autograd functions, which call the forward and backward pass CUDA kernels. If you like to dig deeper on those, you'd have to refer to the CUDA kernel.

I got it, thank you so much!