search

NVIDIA / cutlass

CUDA Templates for Linear Algebra Subroutines
Other
5.76k stars 988 forks source link

[QST] GEMM Epilogue Fusion: Element-wise Ops and Two-Tensor Element-wise Multiplication #1563

Open HanGuo97 opened 6 months ago

HanGuo97 commented 6 months ago

What is your question? Hi, I'd like to compute the following

D = f( matmul(A, B) ) * C

where f is an element-wise activation function, and C is a tensor of the same shape as the output D. In CUTLASS, it seems like only per-dimension scaling is supported in epilogue fusion, but not element-wise scaling between two tensors. In light of this, is such fusion profitable? If so, what are the best ways to implement this?

Thanks in advance for your time!

thakkarV commented 6 months ago

That code is for 2.x api. Which architectures are you interested in?

HanGuo97 commented 6 months ago

I'm looking at Ampere chips.

thakkarV commented 6 months ago

@hwu36 can correct me if I'm wrong but 2.x already support elementwise functions via both the thread epilogue ops and epilogue visitor tree.

HanGuo97 commented 6 months ago

I believe element-wise function is supported, but I was wondering whether element-wise multiplication with another tensor is supported. AFAIK, element-wise multiplication with another scalar or another vector is supported, but not with another tensor.

When I wrote "element-wise multiplication" with another tensor, I meant:

D[i, j] = X[i, j] * C[i,j]

where X is the output of GEMM.

hwu36 commented 6 months ago

You can write your own via epilogue visitor tree (evt). We have an example of using evt.

HanGuo97 commented 6 months ago

Thanks for the quick response! A few quick questions:

  1. Is such fusion "profitable" (element-wise activation, and another element-wise multiplication with a different tensor)? I'd imagine this is a somewhat common epilogue. Given that it's not already in the CUTLASS repo, is it because it's not very useful in practice?

  2. This is a very naive question but what is EVT? I've seen this term at many places but I never understand what it is.

Hongbosherlock commented 6 months ago

Thanks for the quick response! A few quick questions:

  1. Is such fusion "profitable" (element-wise activation, and another element-wise multiplication with a different tensor)? I'd imagine this is a somewhat common epilogue. Given that it's not already in the CUTLASS repo, is it because it's not very useful in practice?
  2. This is a very naive question but what is EVT? I've seen this term at many places but I never understand what it is.

You can learn about EVT here. I'm working on GEMM Epilogue Fusion too, and trying to make it done with EVT. We can work on it together.

shairoz commented 5 months ago

Hi @HanGuo97 , were you able to create the element-wise multiplication epilogue using EVT? I am trying something similar myself and couldn't find examples of using index operations with EVT or epilogues in general, would appreciate your help

github-actions[bot] commented 4 months ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] commented 1 month ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.