Device runs out of memory when using kinect2.

[mklingen]

I've managed to train an object model and run simtrack with a kinect2, but the system hangs for a while and then reports:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Device1D::Device1D: out of memory

tracing in gdb shows that this happens inside callOneCb, presumably in an image callback. Here's the contents of my launch file (I tried switching from hd to qhd to see if that eliminated the error, but it did not.):

<launch>

  <rosparam command="load" file="$(find simtrack_nodes)/config/parameters.yaml"/>
  <param name="simtrack/use_compressed_streams" value="false" />
  <param name="simtrack/tracker/color_only_mode" value="false" />

  <rosparam command="load" file="$(find simtrack_nodes)/config/demo_objects.yaml"/>
  <param name="simtrack/model_path" value="$(find simtrack_nodes)/../data/object_models"/>

<!--  <arg name="stream" value="hd" />
  <node name="simtrack" pkg="simtrack_nodes" type="multi_rigid_node" output="screen">
  <remap from="rgb" to="/head/kinect2/$(arg stream)/image_mono_rect" />
  <remap from="rgb/compressed" to="/kinect2/$(arg stream)/image_mono_rect/compressed" />
  <remap from="rgb_info" to="/head/kinect2/$(arg stream)/camera_info" />
  <remap from="depth" to="/head/kinect2/$(arg stream)/image_depth_rect" />
  <remap from="depth/compressed" to="/head/kinect2/$(arg stream)/image_depth_rect/compressed" />
  </node>-->

  <!-- mixed stream, hd for detection but will be rescaled to qhd for tracking -->
  <node name="simtrack" pkg="simtrack_nodes" type="multi_rigid_node" output="screen" launch-prefix="xterm -e gdb --args">
  <remap from="rgb" to="/head/kinect2/qhd/image_mono_rect" />
  <remap from="rgb/compressed" to="/head/kinect2/qhd/image_mono_rect/compressed" />
  <remap from="rgb_info" to="/head/kinect2/qhd/camera_info" />
  <remap from="depth" to="/head/kinect2/qhd/image_depth_rect" />
  <remap from="depth/compressed" to="/head/kinect2/qhd/image_depth_rect/compressed" />
  </node>

</launch>

The object was generated from this file: https://raw.githubusercontent.com/personalrobotics/pr-ordata/master/data/objects/fuze_bottle_visual.dae

I found that it could create the model using the .dae file, but it also worked when I converted it to an .obj file.

[mklingen]

I switched to a machine with a more beefy GPU and did not encounter this issue. Here's our performance btw!

https://www.youtube.com/watch?v=1yXmpHlNyrk

[karlpauwels]

That looks great! Amazing how fast you got it working!

It does use a lot of memory, especially in full HD mode with the kinect 2. Here's the output of nvidia-smi when I'm detecting and tracking the three example objects. Ideally you should use two GPUs, but they don't have to be super-powerful. Here the first Titan is tracking and running at 52%. It is critical for smooth operation! The second Titan is almost fully utilized for detection but this is not so important .. it only effects the time to recover when tracking is lost. I'll add some of this in the documentation, together with an example video.

+------------------------------------------------------+                       
| NVIDIA-SMI 352.41     Driver Version: 352.41         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TITAN   Off  | 0000:01:00.0      On |                  N/A |
| 39%   61C    P0   110W / 250W |   1087MiB /  6140MiB |     52%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TITAN   Off  | 0000:02:00.0     Off |                  N/A |
| 40%   64C    P0   168W / 250W |   2284MiB /  6143MiB |     84%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1752    G   /usr/bin/X                                     290MiB |
|    0      2545    G   compiz                                         163MiB |
|    0      7922    G   /home/karl/qtcreator-3.2.2/bin/qtcreator        37MiB |
|    0     17805    C   /opt/ros/indigo/lib/nodelet/nodelet            179MiB |
|    0     17939  C+G   ...devel/lib/simtrack_nodes/multi_rigid_node   395MiB |
|    1     17805    C   /opt/ros/indigo/lib/nodelet/nodelet            100MiB |
|    1     17939    C   ...devel/lib/simtrack_nodes/multi_rigid_node  2165MiB |
+-----------------------------------------------------------------------------+

[mklingen]

I'll try to convince my advisor to spring on yet another big GPU ;) In the meantime I wonder if the minimum memory requirements should be listed on the readme (my 860M ran out of VRAM completely).

[karlpauwels]

You never have enough GPUs :) I added the requirements in the documentation.

[davizinho5]

Hello,

I have a similar problem as mklingen, the device runs out of memory most of the times (95%) I launch it, but in this case with an ASUS camera. I thought it was because I use an old graphic card, still is Fermi class , so I though it would be enough. Here is my nvidia-smi:

I think it should have enough memory for both tracking and detection, but anyways, is there a way to use less memory? What if I use a 3D model with less points on it? Also, in my case I would only need the tracking system and I can provide a different detection procedure. Is there a simple way to launch only the tracking and sharing the detection info from another node?

Thanks!