Deep Learning Accelerator

NVIDIA DLA hardware is a fixed-function accelerator engine targeted for deep learning operations. It’s designed to do full hardware acceleration of convolutional neural networks, supporting various layers such as convolution, deconvolution, fully connected, activation, pooling, batch normalization, and others. NVIDIA’s Orin SoCs feature up to two second-generation DLAs while Xavier SoCs feature up to two first-generation DLAs.

DLA software consists of the DLA compiler and the DLA runtime stack. The offline compiler translates the neural network graph into a DLA loadable binary and can be invoked using NVIDIA TensorRT™, NvMedia-DLA or cuDLA. The runtime stack consists of the DLA firmware, kernel mode driver, and user mode driver.

Reference for more details: DLA product page

Why is DLA essential on Orin?

• DLA peak performance contributes between 38% and 74% to Orin's total Deep Learning (DL) performance (depending on the power mode, see table below)
• DLA is on average 3x to 5x more power-efficient than the GPU (depending on the power mode and the workload, see DLA Performance per Watt (Power Efficiency) for details)

Here the distribution of DL TOPs between GPU and DLA on a Jetson AGX Orin 64GB, depending on the power mode:

Note:

• The DLA TOPs of the 30W & 50W power modes on Jetson AGX Orin 64GB are comparable to the maximum clocks on DRIVE Orin platforms for Automotive
• The maximum DLA TOPs on Jetson Orin NX 16GB are comparable to the 15W power mode on Jetson AGX Orin 64GB

DLA Reference Models

In this repo, we will cover a few key DLA-related metrics for standard deep learning model architectures in the context of common reference application implementations.

The goal is to provide a reference baseline about network architectures and how they map to DLA as well as the INT8 accuracy of these networks.

Accuracy on DLA and GPU

*Accuracy measured internally by NVIDIA, there may be slight differences compared to previous MLPerf Inference submissions.

Key takeaways:

• Networks tend to have common network backbones and some variation at the end. You can run the compute-intensive backbones on DLA and final layers for post-processing on the GPU.
• GPU and DLA do not produce bitwise identical results. So the difference in the math is expected and the difference would be within a certain acceptable % of the FP32 reference.

More resources:

• MLPerf Inference 3.1 (Orin DLA)
• MLPerf Inference 3.0 (Orin DLA)
• MLPerf Inference 2.1 (Orin DLA)
• MLPerf Inference 2.0 (Xavier & Orin DLA)

Structured Sparsity Case Study: Object Detection Accuracy with RetinaNet ResNet-34

DLA on the NVIDIA Orin platform supports Structured Sparsity that offers the opportunity to minimize latency and maximize throughput in production. See the TensorRT documentation for details (note that the listed restrictions may not apply anymore to most recent DLA releases).

Below case study presents that training models for Structured Sparsity is expected to maintain accuracy:

Orin DLA Performance

DLA Dense performance

2x DLA images per second on a Jetson AGX Orin 64GB in dense operation measured with JetPack 5.1.1, depending on the power mode:

Key takeaways:

• DLA's peak performance for these models in MAXN power mode is close to the achieved performance with 50W and 30W power modes - even at a reduced power budget, your DLA-based inference pipeline can sustain high throughput.
• Attaching a RetinaNet head to a ResNet-34 backbone adds a considerable number of theoretical MAC operations per inference, and is hence expected to result in lower end-to-end throughput. Choose your task-specific head that you attach to backbones carefully.
• Even at the lowest power mode, all of the benchmarks with high input resolution and batch size 1 can still sustain 30 images per second on 2 DLAs combined.

DLA Sparse performance

Generally, Structured Sparsity shows perf improvements over dense operation for dense Convolution layers that are already math-bound. The more math-bound a layer in dense operation, the higher the expected dense->sparse speedup after applying a 2:4 sparsity pattern.

2x DLA images per second on a Jetson AGX Orin 64GB in sparse operation measured with JetPack 5.1.1, depending on the power mode (with dense->sparse speedup):

Just add --sparsity=force to the trtexec commands from scripts/prepare_models/README.md to reproduce.

Key takeaways:

• Decreasing the power budget on Jetson Orin generally implies reducing the ratio of DLA TOPs to DRAM bandwidth (the hardware's op per Byte ratio). Lowering the hardware's op per Byte ratio means that more layers get the chance to become math-bound, and hence we can also observe higher dense->sparse speedups with lower power modes.

DLA Performance per Watt (Power Efficiency)

DLA & GPU form your perfect team for Deep Learning inference on the SoC. While the GPU delivers the most TOPs in high-power profiles, DLA excels at power efficiency.

Below table shows the Perf/W ratio of DLA relative to the GPU (accelerator power only, perf metric: images per second) on a Jetson AGX Orin 64GB measured with JetPack 5.1.1, depending on the power mode:

Key takeaways:

• DLA is about 3x to 5x more power efficient than GPU for these benchmarks
• Enabling Structured Sparsity generally improves DLA's power efficiency
• At the lowest power mode of 15W, DLA's power efficiency is the highest (where 74% total Orin peak DL INT8 performance comes from the DLAs)

ONNX operators supported on DLA

See operators/README.md for details on ONNX operators already supported on DLA and planned to be supported in future releases.

Setup

Install the Python dependencies with (only supported on x86 hosts):

python3 -m pip install requirements.txt