Benchmark on Firefly-RK3399

[psyhtest]

Now that we have the latest Firefly-RK3399 BSP (xubuntu1604_201711301130), we should record benchmarking results on this platform in the CK format.

[psyhtest]

As I understand from the README, we have three programs all using the same command key:

$ ck run program:tvm-arm  --cmd_key=all
$ ck run program:tvm-nnvm  --cmd_key=all
$ ck run program:tvm-mxnet --cmd_key=all

How about we perform and record the experiments as follows:

$ ck benchmark program:tvm-<program name> \
--cmd_key=all --repetitions=10 \
--record --record_repo=local \
--record_uoa=ck-request-asplos18-tvm-<program name>-<model name>-<precision> \ 
--tags=request,request-asplos18,tvm,<program name>,<model name>,v<precision>

where:

• is one of `arm`, `nnvm`, `mxnet`;
• is one of `mobilenets`, `alexnet`, `vgg16`;
• is one of `fp32`, `fp16`. (I'm writing this from memory so please correct me if I'm wrong.)

[gfursin]

Looks ok to me ;) ! Thanks! Just before taking off ;) ...

[fvella]

Tvm the support

model [mobilenet || resnet18 || vgg16] 
dtype=[float16 || float32]

ARM

backend[cl || neon]
model[mobilenet || vgg16] 
conv_method[gemm || direct] 
dtype[float16 || float32]"

Mxnet

model=[mobilenet || resnet18 || vgg16] 

[fvella]

List of Experiments:

<name><backend><model><data_type>

MXNET

mxnet-openblas-mobilenet-gemm-float32
mxnet-openblas-resnet18-gemm-float32
mxnet-openblas-vgg16-gemm-float32

TVM

tvm-opencl-mobilenet-tvm-mali-float16
tvm-opencl-mobilenet-tvm-mali-float32
tvm-opencl-resnet18-tvm-mali-float16
tvm-opencl-resnet18-tvm-mali-float32
tvm-opencl-vgg16-tvm-mali-float16
tvm-opencl-vgg16-tvm-mali-float32

ARM

armcl-cl-mobilenet-direct-float16
armcl-cl-mobilenet-direct-float32
armcl-cl-mobilenet-gemm-float16
armcl-cl-mobilenet-gemm-float32
armcl-cl-vgg16-direct-float16
armcl-cl-vgg16-direct-float32
armcl-cl-vgg16-gemm-float16
armcl-cl-vgg16-gemm-float32
armcl-neon-mobilenet-direct-float16
armcl-neon-mobilenet-direct-float32
armcl-neon-mobilenet-gemm-float16
armcl-neon-mobilenet-gemm-float32
armcl-neon-vgg16-direct-float16
armcl-neon-vgg16-direct-float32
armcl-neon-vgg16-gemm-float16
armcl-neon-vgg16-gemm-float32

[psyhtest]

Can you please attach here a zip of the experimental data when it's ready?

[fvella]

ARM experiment had some problem. I have to re-run just them.

[fvella]

I have the results. WRT to the list posted before we have less experiments. The application does not allow to run some of them. i.e. neon with fp16..

[fvella]

@gfursin request-tvm-firefly.zip \cc @psyhtest

[gfursin]

Cool! Thanks a lot @fvella - I am now preparing presentations for tomorrow and if I find time, I will check it tonight ...

[gfursin]

I unified all experiments and recorded them in the CK format.