search

pytorch / glow

Compiler for Neural Network hardware accelerators
Apache License 2.0
3.22k stars 689 forks source link

[Tensor Cores] Does glow OpenCL backend support NVIDIA Tensor Cores? #3949

Open XiZiler opened 4 years ago

XiZiler commented 4 years ago

I'm new to Glow and want to utilize my NVIDIA GPU's Tensor Cores which is specialized for DNNs. The official demos of Glow offer an OpenCL backend to run networks on GPU, but not Tensor Cores. I'm wondering if there is a easy way to support Tensor Cores for higher efficiency?

nickgg commented 4 years ago

Hi @XiZiler. Currently Glow supports only OpenCL for running networks on GPUs, which I believe works just fine on a Tensor Core architecture, but may not make the most efficient use of the hardware. If you want something more efficient you'd need an NV specific Glow backend. Are you interested in contributing to such a backend?

mratsim commented 4 years ago

You can use PTX inline assembly in OpenCL. You don't need to write a whole new backend, just specialize the operations accelerated by tensor cores (i.e. matrix multiplications).

There are actually extensions to ClBlast for float16 GEMM on Tensor Cores in the various LeelaChess and LeelaZero projects (the license is GPL though).