Determine and Implement Efficient Visual Processing Pipeline

[yameatmeyourdead]

Decide between/conjunction of:

• OpenCV

  • Steep learning curve
  • Need already-trained models
  • opencv with cudnn backend already built on docker image

• Tensorflow

  • slower for training
  • less memory usage
  • harder to setup
  • FOSS
  • Not an SDK, large learning curve

• Deepstream

  • Super optimized for jetsons
  • Proprietary NVidia tech (not FOSS)
  • SDK with decent documentation

• Torch (fast for training, more memory usage, easier to setup)

• Keras (very slow but easily readable)

[Hermanoid]

Deepstream is what my heart initially goes for, especially if we could interop it with something like Keras for ease of setup. Just how much slower is Keras (with Tensorflow backend) than Torch? I've had a beautiful experience with Keras for training, and it's fairly GPU accelerated for training. Also... Torch looks tres difficile to program with 😬

[yameatmeyourdead]

Keras, being much more high-level, suffers from the level of abstraction necessary to make neural nets not as convoluted. This results in a consistently higher memory usage and worse performance. These losses in efficiency may be outweighed by the ease of experimentation and reduced development time Keras provides. Keras is noticeably slower. My main concern with Keras is the reduced level of customization and debugging may cause roadblocks for more advanced tasks in the future. Perhaps the larger learning curves of Torch/Tensorflow would provide more useful insight into potential optimizations and improvements through a deeper understanding of how the models work.

If we use Deepstream, we should eventually use/learn Tensorflow and TensorRT. Personally, I think we should bite the bullet and go for Deepstream with Tensorflow and TensorRT.

From some quick google searches it looks possible (though not a walk in the park) to convert keras models to deepstream-compatible models. Perhaps a hybrid use of keras where performance can suck and deepstream if we need faster real-time models will serve us well (similar to our current use of python and c++).

[Hermanoid]

I definitely did not expect Keras to be so dramatically slower.

We'll need to think about what kind of tasks we'll need to process for... for super-duper precise, high-end obstacle finding, we might benefit from lower-level programming (like what you get with pytorch/tensorflow), but from my understanding of last year's challenge, the tasks are quite simple, to the point that we might not even need Machine Learning in the first place (we could get by with some blob/line detection, etc). This makes me think that Keras's degree of control would be plenty. As for speed, I agree that we might need to strike a trade-off between running speed and development difficulty depending on the task. Since I suspect that we will only need Machine Learning for like one task to classify an object or something of the sort, I think it would be enough to run a Keras model straight on the GPU, and not even mess with Deepstream acceleration. We'd have to do this on the basestation TX2. That's not hard, since we've got full ROS connectivity between the ROV and Basestation, but it will slightly increase latency.

On Fri, Jul 8, 2022 at 11:03 PM Zac Stanton @.***> wrote:

Keras, being much more high-level, suffers from the level of abstraction necessary to make neural nets not as convoluted. This results in a consistently higher memory usage and worse performance. These losses in efficiency may be outweighed by the ease of experimentation and reduced development time Keras provides. Keras is noticeably slower https://deepsense.ai/wp-content/uploads/2018/06/plot11.png.pagespeed.ce.amL3pDsEAG.png. My main concern with Keras is the reduced level of customization and debugging may cause roadblocks for more advanced tasks in the future. Perhaps the larger learning curves of Torch/Tensorflow would provide more useful insight into potential optimizations and improvements through a deeper understanding of how the models work.

If we use Deepstream, we should eventually use/learn Tensorflow and TensorRT. Personally, I think we should bite the bullet and go for Deepstream with Tensorflow and TensorRT.

From some quick google searches it looks possible (though not a walk in the park) to convert keras models to deepstream-compatible models. Perhaps a hybrid use of keras where performance can suck and deepstream if we need faster real-time models will serve us well (similar to our current use of python and c++).

— Reply to this email directly, view it on GitHub https://github.com/Colorado-School-of-Mines-Robotics-Club/MATE_ROV_Programming/issues/6#issuecomment-1179473927, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIV7VHKRTKIJC6SGVJUWFDVTD2XHANCNFSM527K6Z4Q . You are receiving this because you commented.Message ID: <Colorado-School-of-Mines-Robotics-Club/MATE_ROV_Programming/issues/6/1179473927 @github.com>