NVIDIA-ISAAC-ROS / isaac_ros_dnn_stereo_depth

NVIDIA-accelerated, deep learned stereo disparity estimation
https://developer.nvidia.com/isaac-ros-gems
Apache License 2.0
92 stars 10 forks source link

Using TAO Toolkit for training/pruning #7

Closed chen-harrison closed 1 year ago

chen-harrison commented 1 year ago

While looking through Nvidia's NGC catalog where the ESS DNN resides, I started reading about the TAO Toolkit and the ability to take pre-trained models and augment them with additional data, perform pruning, and other such operations in order to fully optimize them.

Image from site

In my desired use case, I'd like to train further on a custom dataset, one that is captured on the stereo camera hardware I intend to later perform inference on. In theory, this would allow the network to perform even better. The ability to prune and thus further decrease the computational cost is also appealing.

My problem is that I'm having a difficult time reading about whether this is possible with .etlt files. There are Jupyter notebooks available to perform this type of work on more commonly used deep learning tasks (object detection, semantic segmentation), and while there isn't one for the ESS DNN, I'm wondering if it's even possible based on resources currently available, and if not, if that is something that intends to be supported in the future.

Thanks!

gordongrigor commented 1 year ago

ESS is a custom DNN which is not supported for transfer learning, and pruning in TAO.

gordongrigor commented 1 year ago

Closing.