search

issamemari / model_optimization

Playground for testing model optimization techniques
1 stars 0 forks source link

Fast Feedforward is slower #1

Open superFilicos opened 10 months ago

superFilicos commented 10 months ago

Hi! I'm also focusing on the project Fast feedforward, and when I use it in my model I found that it was much slower than Linear. I also use your code and get the same result. image So, did you encounter the same problem with me?

issamemari commented 10 months ago

Indeed I also get similar results. I noticed however that you can get a speedup in extreme cases. For example under the following configuration:

in_features = 2000
leaf_features = 5
out_features = 2000
depth = 4
batch_size = 500
max_iter = 1000

I get ~1.7x speedup:

FastFeedForward avg batch inference duration: 0.027593517475409817
Linear avg batch inference duration: 0.04902171762295078
Speedup: 1.7765664586487342
FastFeedForward avg batch inference duration: 0.027575688951612876
Linear avg batch inference duration: 0.04907984716129028
Speedup: 1.7798230625320297
FastFeedForward avg batch inference duration: 0.027542462952380928
Linear avg batch inference duration: 0.04911540638095234
Speedup: 1.7832612306992872
superFilicos commented 10 months ago

Hi! Yes, but you need to set the hidden_size to more than 2000... I wish maybe someday I can have such powerful gpus...