AutoRT for Device Runtime:
AutoRT is a compiler solution that helps runtime users to invent, benchmark and optimize operators for Pytorch using your own accelerators:
Platform | OS Requirement | Python Requirement | Download Link |
---|---|---|---|
DirectX 12 (x86_64) | Windows >= 10 / Microsoft XBox | Python3.12 (Windows) | python3.12 -m pip install https://github.com/microsoft/antares/releases/download/v0.9.6/autort-0.9.6.3+directx.win-cp312-cp312-win_amd64.whl |
Vulkan 1.3 (x86_64) | Ubuntu >= 18.04 | Python3.12 (Linux) | python3.12 -m pip install https://github.com/microsoft/antares/releases/download/v0.9.6/autort-0.9.6.3+vulkan.linux-cp312-cp312-manylinux1_x86_64.whl |
CUDA >= 11.0 (x86_64) | Windows >= 10 / Ubuntu >= 18.04 | Python 3.10/3.12 | python3 -m pip install https://github.com/microsoft/antares/releases/download/v0.9.6/autort-0.9.6.3.4+cuda.zip |
CUDA >= 12.0 (aarch64) | Ubuntu >= 22.04 | Python 3.10/3.12 | python3 -m pip install https://github.com/microsoft/antares/releases/download/v0.9.6/autort-0.9.6.3.4+cuda.zip |
.. | .. | .. | .. (More coming soon) .. |
For CUDA, here are several Ubuntu >= 18.04 equivalent containers below:
$ python.exe -m autort.benchmark.memtest
...
[1000/1000] AutoRT Device Memory Bandwidth: (Actual ~= 468.12 GB/s) (Theoretical ~= 561.75 GB/s)
$ python.exe -m autort.benchmark.fp32test
...
[5000/5000] AutoRT FP32 TFLOPS: (Actual ~= 9.84 TFLOPS) (Theoretical ~= 10.93 TFLOPS)
$ python.exe -m autort.benchmark.mmtest
...
$ python.exe -m autort.benchmark.copytest
...
$ python.exe -m autort.benchmark.launchtest
...
>> import torch, autort
>> data = torch.arange(0, 10, dtype=torch.float32, device=autort.device())
f = autort.export(ir="sigmoid_f32[N] = 1 - 1 / (1 + data[N].call(strs.exp))", inputs=["data=float32[N:4096000]"], config="tune:5") print(f(data)) tensor([0.5000, 0.7311, 0.8808, 0.9526, 0.9820, 0.9933, 0.9975, 0.9991, 0.9997, 0.9999]) print(autort.ops.sigmoid_f32(data)) tensor([0.5000, 0.7311, 0.8808, 0.9526, 0.9820, 0.9933, 0.9975, 0.9991, 0.9997, 0.9999])
# Fist, create a custom sigmoid activation operator with 5 tuning steps:
$ autort --ir "sigmoid_f32[N] = 1 - 1 / (1 + data[N].call(strs.exp))" -i data=float32[N:4096000] -c "tune:5"
$ python.exe
import torch, autort
data = torch.arange(0, 10, dtype=torch.float32, device=autort.device()) output = autort.ops.sigmoid_f32(data) print(output) tensor([0.5000, 0.7311, 0.8808, 0.9526, 0.9820, 0.9933, 0.9975, 0.9991, 0.9997, 0.9999]) output = torch.nn.functional.sigmoid(data) print(output) tensor([0.5000, 0.7311, 0.8808, 0.9526, 0.9820, 0.9933, 0.9975, 0.9991, 0.9997, 0.9999])
$ python.exe -m autort.examples.01_sort_even_first
Input : tensor([101, 102, 208, 99, 1, 127, 62, 8, 336, 336], dtype=torch.int32)
(is_even) tensor([False, True, True, False, False, False, True, True, True, True])
Output: tensor([102, 208, 62, 8, 336, 336, 101, 99, 1, 127], dtype=torch.int32)
(is_even) tensor([ True, True, True, True, True, True, False, False, False, False])
$ python.exe -m autort.examples.02_mnist
...
step = 800, loss = 0.5159, accuracy = 87.50 %
step = 900, loss = 0.5511, accuracy = 84.38 %
step = 1000, loss = 0.2616, accuracy = 93.75 %
...
$ python.exe -m autort.examples.03_llama_tiny
What is that?"
"That is the sun," her mom said. "It gives us heat."
The little girl was amazed. She had never seen the heat before.
"Can we go outside and feel the sun?" she asked.
"Yes," her mother said.
...
$ python.exe -m autort.examples.05_llama2_7b_int4
How large is Atlantic Ocean?
The Atlantic Ocean is the second largest ocean on Earth, covering approximately 20% of the Earth's surface. ...
$ python.exe -m autort.examples.06_diffuser_no_opt
...
Image converted from `./samurai_nn.png` to `samurai_nn_diffused.png`..
If you like it, welcome to report issues or donate stars which can encourage AutoRT to support more backends, more OS-type and more documentations. See More Information about Microsoft Contributing and Trademarks.