Can input channel be different from output channel？

[jimmylihui]

Thanks for your working, I saw in all examples, input channels are all equal to output channels. Can input channels be different from output channels?

[DanFu09]

We currently do not support this. We currently only support "depthwise" convolutions (or SISO convolutions if you come from the signal processing literature). These treat each channel in the input independently for the convolution.

Do you have use cases for input channels != output channels for long convolutions? Would you mind sharing a simple example in PyTorch so we can understand exactly what the behavior/spec is?

[jimmylihui]

For conv1d in PyTorch. There is input channel and output channel numbers. And it is useful for downsampling and upsampling in like Unet

[DanFu09]

Ah I see. My impression is that most of those are grouped convolutions, so you have a full dense layer connecting all the input and output channels for each filter (confusingly, in PyTorch Conv1D, this is called with groups=1).

For small kernels, I think the PyTorch/cuDNN code is pretty decent - do you know of examples where there are very long kernels in this case? It's something we've been curious about, so if there are example architectures to benchmark/test out that would be great for us!

[jimmylihui]

Because like Unet in diffusion model is full of convolution. I am curious about if your design can accelerate it.

[DanFu09]

This is actually something we're working on... we may have some good news soon, stay tuned :)

[jimmylihui]

sad. I guess I need another idea now. By the way, how long kernels you define is large?

[DanFu09]

If you reach out via email, happy to discuss our ideas further!

how long kernels you define is large?

We've seen two regimes -- 1/3/5/7, and then kernels as long as the input sequence (like input is 256, kernel is 256). We usually refer to this latter case as long convolutions.

[arnavdantuluri]

If you reach out via email, happy to discuss our ideas further!

Mind if I reach out as well? Been working on a diffusion acceleration project and would love to share thoughts and ideas with you guys!

[DanFu09]

Please do!

On Wed, Nov 29, 2023 at 8:11 PM AandD @.***> wrote:

If you reach out via email, happy to discuss our ideas further!

Mind if I reach out as well? Been working on a diffusion acceleration project and would love to share thoughts and ideas with you guys!

— Reply to this email directly, view it on GitHub https://github.com/HazyResearch/flash-fft-conv/issues/2#issuecomment-1833078360, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDDIIVSH4NE6MSASXTKLRDYHABQXAVCNFSM6AAAAAA7M2QO42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTGA3TQMZWGA . You are receiving this because you modified the open/close state.Message ID: @.***>

[arnavdantuluri]

Please do!

Shot you guys an email!

[arnavdantuluri]

@DanFu09 Just following up with you on this - I sent you an email yesterday about connecting. Let me know if you didn't receive it!