search

autumnai / collenchyma

Extendable HPC-Framework for CUDA, OpenCL and common CPU
http://autumnai.github.io/collenchyma
Apache License 2.0
475 stars 33 forks source link

Release numbers for the benchmarks againts other frameworks #44

Closed botev closed 8 years ago

botev commented 8 years ago

Hi, I watched the talk on Rust Berlin, however I did not find any place where there are actually published numbers about the speed performance of the framework. Also on the talk it was said that it is twice faster than Tensorflow and Caffee, however it was not specified on which device, as well as I did not hear comments for comparison with Torch, Theano and MXNet which are defacto the current lead in performance in ML. It would be really great if there could be something that we can actually see as numbers as well as the code for the other frameworks against which you are benchmarking.

hobofan commented 8 years ago

We mostly benchmarked the popular convnet-benchmarks.

We have some numbers that we got by running it on our local machine here: https://github.com/autumnai/leaf/tree/dump-0.2.0#benchmarks (code is under example/benchmarks.rs). The tests were run on the GPU with cuDNNv3, similar to the other frameworks. The machine contains a Intel i7-2600k and a NVIDIA Titan X.

We realize the the benchmarks are not perfect yet, and we will create a PR to the convnet-benchmarks repo after the release Leaf 0.2, so it can be properly peer-reviewed and representative numbers can be produced.

It would also be great to have other benchmarks, since I think there are some use-cases where I think Leaf can be significantly faster e.g. one-off inference and network creation time.

There is a issue (https://github.com/autumnai/leaf/issues/26) dedicated for tracking cross-framework benchmarks, but I just saw that it is completely empty :sweat_smile:

EDIT:

Since I don't know how long that branch will stay up and I am not sure if that will stay in the README for master here are the numbers:

Benchmarks

(in ms) Leaf Torch Neon Caffe Tensorflow
Alexnet
FORWARD 30.8 33.0 30.9 42.0 46.1
BACKWARD 70.6 66.1 67.2 85.3 156.0
TOTAL 101.4 99.1 98.1 127.3 202.1
Overfeat
FORWARD 104.6 113.5 142.3
BACKWARD 216.7 213.7 287.9
TOTAL 321.3 327.2 430.2
hobofan commented 8 years ago

@Botev Do you have any further questions? Otherwise I would close the issue (there still is autumnai/leaf#26).

hobofan commented 8 years ago

We now have benchmarks at https://github.com/autumnai/deep-learning-benchmarks .