Low VRAM Settings - Githubissues

rocketmax commented 4 years ago

Hello,

I know that I'm pushing my luck trying to run this on a laptop with a GTX 960M (2 GB), but I'm fairly limited in my hardware right now. The Openpose demo will run if I lower the net_resolution parameter down below -1x224. I've tried to run this with that parameter added to the openpose_args line in the launch file:

<arg name="openpose_args" value=" --model_folder /home/ubuntafoo/openpose/models/ --net_resolution -1x160 --model_pose BODY_25" />

I've even tried to lower it down to -1x16 and that didn't help, so I'm wondering if there is something within the system that locks the network resolution or if there is any way to lower the graphics memory needed that you know of.

Thanks for any help you can provide!

Here's my full error log

roslaunch ros_openpose run_realsense.launch
... logging to /home/ubuntafoo/.ros/log/d8509af4-42e6-11ea-9c31-441ca8e2c5e3/roslaunch-ubuntafoo-XPS-15-9550-2683.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ubuntafoo-XPS-15-9550:35019/

SUMMARY
========

PARAMETERS
 * /camera/realsense2_camera/accel_fps: 250
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: True
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/allow_no_texture_points: False
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: -2.0
 * /camera/realsense2_camera/color_fps: 30
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 480
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 640
 * /camera/realsense2_camera/depth_fps: 30
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 480
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 640
 * /camera/realsense2_camera/device_type: 
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: True
 * /camera/realsense2_camera/enable_depth: True
 * /camera/realsense2_camera/enable_fisheye1: True
 * /camera/realsense2_camera/enable_fisheye2: True
 * /camera/realsense2_camera/enable_fisheye: True
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: True
 * /camera/realsense2_camera/enable_infra2: True
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_sync: False
 * /camera/realsense2_camera/filters: pointcloud
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: 30
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: 480
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: 640
 * /camera/realsense2_camera/gyro_fps: 400
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/imu_optical_frame_id: camera_imu_optica...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 30
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_width: 640
 * /camera/realsense2_camera/initial_reset: False
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: True
 * /camera/realsense2_camera/publish_tf: True
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/tf_publish_rate: 0.0
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: 
 * /camera/realsense2_camera/usb_port_id: 
 * /rosOpenpose/cam_info_topic: /camera/color/cam...
 * /rosOpenpose/color_topic: /camera/color/ima...
 * /rosOpenpose/depth_topic: /camera/aligned_d...
 * /rosOpenpose/frame_id: camera_depth_opti...
 * /rosOpenpose/openpose_model_dir: /home/ubuntafoo/o...
 * /rosOpenpose/pub_topic: /frame
 * /rosdistro: melodic
 * /rosversion: 1.14.3
 * /visualizer/frame_id: camera_depth_opti...
 * /visualizer/id_text_offset: -0.05
 * /visualizer/id_text_size: 0.2
 * /visualizer/pub_topic: /frame
 * /visualizer/skeleton_hands: True
 * /visualizer/skeleton_line_width: 0.01

NODES
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)
  /
    rosOpenpose (ros_openpose/rosOpenpose)
    rviz (rviz/rviz)
    visualizer (ros_openpose/visualizer.py)

auto-starting new master
process[master]: started with pid [2705]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to d8509af4-42e6-11ea-9c31-441ca8e2c5e3
process[rosout-1]: started with pid [2716]
started core service [/rosout]
process[rosOpenpose-2]: started with pid [2719]
process[visualizer-3]: started with pid [2720]
process[camera/realsense2_camera_manager-4]: started with pid [2725]
process[camera/realsense2_camera-5]: started with pid [2726]
process[rviz-6]: started with pid [2727]
[ INFO] [1580336983.675725044]: Initializing nodelet with 4 worker threads.
[ INFO] [1580336984.001942148]: Starting ros_openpose...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
[ WARN] [1580336984.029283933]: Empty color image frame detected. Ignoring...
[ INFO] [1580336984.148588650]: RealSense ROS v2.2.12
[ INFO] [1580336984.148627475]: Running with LibRealSense v2.32.1
[ INFO] [1580336984.158136633]:  
[ INFO] [1580336984.163999724]: Device with serial number 827312072596 was found.

[ INFO] [1580336984.164057985]: Device with physical ID /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/video4linux/video0 was found.
[ INFO] [1580336984.164082199]: Device with name Intel RealSense D435 was found.
[ INFO] [1580336984.166126310]: Device with port number 2-1 was found.
[ INFO] [1580336984.170358664]: getParameters...
[ INFO] [1580336984.211244635]: setupDevice...
[ INFO] [1580336984.211288742]: JSON file is not provided
[ INFO] [1580336984.211310488]: ROS Node Namespace: camera
[ INFO] [1580336984.211332847]: Device Name: Intel RealSense D435
[ INFO] [1580336984.211372704]: Device Serial No: 827312072596
[ INFO] [1580336984.211411614]: Device physical port: /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/video4linux/video0
[ INFO] [1580336984.211456552]: Device FW version: 05.12.02.100
[ INFO] [1580336984.211488818]: Device Product ID: 0x0B07
[ INFO] [1580336984.211520449]: Enable PointCloud: On
[ INFO] [1580336984.211551207]: Align Depth: On
[ INFO] [1580336984.211641564]: Sync Mode: On
[ INFO] [1580336984.211961209]: Device Sensors: 
[ INFO] [1580336984.212180477]: Stereo Module was found.
[ INFO] [1580336984.212244526]: RGB Camera was found.
[ INFO] [1580336984.212296793]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212324887]: (Fisheye, 1) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212337963]: (Fisheye, 2) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212349931]: (Gyro, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212373233]: (Accel, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212389229]: (Pose, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1580336984.212560628]: Add Filter: pointcloud
[ INFO] [1580336984.213314725]: num_filters: 1
[ INFO] [1580336984.213336833]: Setting Dynamic reconfig parameters.
[ INFO] [1580336984.257288313]: Done Setting Dynamic reconfig parameters.
[ INFO] [1580336984.291319657]: depth stream is enabled - width: 640, height: 480, fps: 30, Format: Z16
[ INFO] [1580336984.291645147]: infra1 stream is enabled - width: 640, height: 480, fps: 30, Format: Y8
[ INFO] [1580336984.291880201]: infra2 stream is enabled - width: 640, height: 480, fps: 30, Format: Y8
 29/01 17:29:44,292 WARNING [140157112588032] (backend-v4l2.cpp:1208) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
[ INFO] [1580336984.297269887]: color stream is enabled - width: 640, height: 480, fps: 30, Format: RGB8
[ INFO] [1580336984.299858645]: setupPublishers...
[ INFO] [1580336984.303243485]: Expected frequency for depth = 30.00000
[ INFO] [1580336984.337815966]: Expected frequency for infra1 = 30.00000
[ INFO] [1580336984.355522195]: Expected frequency for aligned_depth_to_infra1 = 30.00000
[ INFO] [1580336984.376763705]: Expected frequency for infra2 = 30.00000
[ INFO] [1580336984.392493888]: Expected frequency for color = 30.00000
[ INFO] [1580336984.408744692]: Expected frequency for aligned_depth_to_color = 30.00000
[ INFO] [1580336984.427550385]: setupStreams...
[ INFO] [1580336984.428643909]: insert Depth to Stereo Module
[ INFO] [1580336984.428828786]: insert Color to RGB Camera
[ INFO] [1580336984.428867799]: insert Infrared to Stereo Module
[ INFO] [1580336984.428895157]: insert Infrared to Stereo Module
[ INFO] [1580336984.486476199]: SELECTED BASE:Depth, 0
[ INFO] [1580336984.545136174]: RealSense Node Is Up!
 29/01 17:29:44,607 WARNING [140156860278528] (ds5-timestamp.cpp:76) UVC metadata payloads not available. Please refer to the installation chapter for details.
[ WARN] [1580336985.269209548]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1580336988.414509582]: No stream match for pointcloud chosen texture Process - Color
F0129 17:29:48.588560  2774 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
    @     0x7f92f154d0cd  google::LogMessage::Fail()
    @     0x7f92f154ef33  google::LogMessage::SendToLog()
    @     0x7f92f154cc28  google::LogMessage::Flush()
    @     0x7f92f154f999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f92f1062e58  caffe::SyncedMemory::mutable_gpu_data()
    @     0x7f92f0ec2272  caffe::Blob<>::mutable_gpu_data()
    @     0x7f92f10d19af  caffe::PoolingLayer<>::Forward_gpu()
    @     0x7f92f1026001  caffe::Net<>::ForwardFromTo()
    @     0x7f92f41f73eb  op::NetCaffe::forwardPass()
    @     0x7f92f42162da  op::PoseExtractorCaffe::forwardPass()
    @     0x7f92f4210715  op::PoseExtractor::forwardPass()
    @     0x7f92f420d6d3  op::WPoseExtractor<>::work()
    @     0x7f92f4247b59  op::Worker<>::checkAndWork()
    @     0x7f92f4247cfb  op::SubThread<>::workTWorkers()
    @     0x7f92f425079b  op::SubThreadQueueInOut<>::work()
    @     0x7f92f424abbb  op::Thread<>::threadFunction()
    @     0x7f92f24739e0  (unknown)
    @     0x7f92f27466db  start_thread
    @     0x7f92f1c9f88f  clone
================================================================================REQUIRED process [rosOpenpose-2] has died!
process has died [pid 2719, exit code -6, cmd /home/ubuntafoo/catkin_ws/devel/lib/ros_openpose/rosOpenpose --hand __name:=rosOpenpose __log:=/home/ubuntafoo/.ros/log/d8509af4-42e6-11ea-9c31-441ca8e2c5e3/rosOpenpose-2.log].
log file: /home/ubuntafoo/.ros/log/d8509af4-42e6-11ea-9c31-441ca8e2c5e3/rosOpenpose-2*.log
Initiating shutdown!
================================================================================
[rviz-6] killing on exit
[camera/realsense2_camera-5] killing on exit
[camera/realsense2_camera_manager-4] killing on exit
[visualizer-3] killing on exit
[rosOpenpose-2] killing on exit
terminate called without an active exception
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

ravijo commented 4 years ago

Hi @rocketmax,

I am sorry to hear about the issue. As you said already, you are using a GTX 960M (2 GB) graphics card. It is evident from the following log message that you are running out of memory-

Check failed: error == cudaSuccess (2 vs. 0)  out of memory

This package, i.e. ros_openpose doesn't use GPU directly. However, it depends on OpenPose which uses GPU heavily.

Since the error is coming from OpenPose software, I suggest you run the openpose software on example data as shown below-

./build/examples/openpose/openpose.bin --video examples/media/video.avi

Now, please try to reduce the --net_resolution and give it a try with --model_pose COCO. Also, please make sure that cuDNN is installed correctly. You need to compile OpenPose with it.

I also suggest you read and try this documentation.

Let me know if this helps!

rocketmax commented 4 years ago

Hey @ravijo that helped a lot! The COCO model does use less memory and works well for me. Now I'm having an issue similar to Issue #4 , but I am going to keep working on it.

Thanks again!

ravijo commented 4 years ago

Hi @rocketmax,

Great! Glad that it worked. Feel free to post your error if you can't fix it.

ravijo / ros_openpose

Low VRAM Settings #6