search

HazyResearch / safari

Convolutions for Sequence Modeling
Apache License 2.0
864 stars 71 forks source link

Long latency when loading fft_conv kernel for the first time #8

Open sylee0124 opened 1 year ago

sylee0124 commented 1 year ago

The first kernel launch of fft_conv_fwd takes abnormally long time (about 100 sec). After the first kernel launch it works fine so for training it's not much of an issue but it makes debugging very cumbersome. Could it be a problem with my ld options?

image

DanFu09 commented 1 year ago

This is something we experience too - it makes it super annoying to debug...

We're looking into it, but if anyone has suggestions we'd love to hear them!