search

Bihaqo / TensorNet

136 stars 37 forks source link

training speed #7

Open WWWJL opened 4 years ago

WWWJL commented 4 years ago

Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you

Bihaqo commented 4 years ago

In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process.

On Thu, 16 Apr 2020 at 07:16, WWWJL notifications@github.com wrote:

Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bihaqo/TensorNet/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ .

Bihaqo commented 4 years ago

Also note that in the MATLAB implementation I wrote the backward pass manually and it's much slower than the speed you can get with autodiff (e.g. by writing it in tensorflow or pytorch)

On Thu, 16 Apr 2020 at 13:52, Alexander Novikov sasha.v.novikov@gmail.com wrote:

In my experience, tensor net topically don't help with the training speed (unless you're going for truly gigantic layers, which would be impractical without tensornet): the time per iteration is usually almost the same (maybe like 10% faster) and you need to do a bit more iterations than when training without tensornet. That's probably because on training a single matmul is rarely the bottleneck of the entire training process.

On Thu, 16 Apr 2020 at 07:16, WWWJL notifications@github.com wrote:

Hello, has TensorNet accelerated during training? Hope you can share the results of the training speed, thank you

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Bihaqo/TensorNet/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK6V2OD2MDGUCXQLU6T76DRM2PDTANCNFSM4MJHV4JQ .