Added polling option for multiple cameras support

[jskhu]

Fixed the issue of multiple cameras being launched from the same file by introducing a polling feature. Now, each node instance polls the Ximea API until it finds its specified camera. Fixes #10.

@carloswanguw

[jskhu]

@mpitropov I've introduced periodic polling which will space the usage of the Ximea API per nodelet which should resolve the nodes stopping each other from connecting. Let me know if the issue persists.

[mpitropov]

I have 8cameras in a launch file and 6 of them are plugged in. Now only a few start. Here is some logging code [ INFO] [1511800513.069231533]: Capture CB Looping... [ INFO] [1511800513.073138434]: Capture CB Looping... [ INFO] [1511800513.083461157]: Capture CB Looping... [ INFO] [1511800513.091189847]: Capture CB Looping... xiAPI: EAL_DeviceEnumerator::OpenDevice - can not find device by ID 49606951 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.105210641]: Polling cam_serialno 49606951 56 32558 [ INFO] [1511800514.105260258]: Capture CB Looping... xiAPI: EAL_DeviceEnumerator::OpenDevice - can not find device by ID 49605551 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.158381028]: Polling cam_serialno 49605551 56 32638 [ INFO] [1511800514.158422988]: Capture CB Looping... xiAPI: EAL_DeviceEnumerator::OpenDevice - can not find device by ID 49605451 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.186615940]: Polling cam_serialno 49605451 56 32557 [ INFO] [1511800514.186664058]: Capture CB Looping... xiAPI: EAL_DeviceEnumerator::OpenDevice - can not find device by ID 50601651 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.201367847]: Polling cam_serialno 50601651 56 32685 [ INFO] [1511800514.201414583]: Capture CB Looping... xiAPI: EAL_IF_xi3API::OpenDeviceHandler failed for id:3 xiAPI: xiFGenTL Fatal error: Expected GC_ERR_SUCCESS in:../Layers/EAL/EAL_IF_xiFAPI_Top.cpp OpenDeviceHandler/Line:61 xiAPI: OpenDevice Error 56 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.204456445]: Polling cam_serialno 49608451 56 32614 xiAPI: XIMEA Camera API V4.15.07.00 xiAPI: Adding camera context: dwID=49608851 ptr=F53B9000 processID=0000504B xiAPI: Create handles 1 Process 0000504B xiAPI: EAL_IF_xi3API::OpenDeviceHandler failed for id:1 xiAPI: xiFGenTL Fatal error: Expected GC_ERR_SUCCESS in:../Layers/EAL/EAL_IF_xiFAPI_Top.cpp OpenDeviceHandler/Line:61 xiAPI: OpenDevice Error 56 xiAPI: xiAPI error: Expected XI_OK in:../API/xiFAPI/interfaces/01_top/xifapi_Top.cpp xiOpenDeviceBy/Line:117 [ INFO] [1511800514.259666436]: Polling cam_serialno 49603051 56 32621 xiAPI: Enable sensor xiAPI: xiReadFileFFS 'SensFPNCorrections' not found xiAPI: ScmCorrectorA::LoadAndParseConfigurationFile Warning: file SensFPNCorrections not found xiAPI: XIMEA Camera API V4.15.07.00 xiAPI: Adding camera context: dwID=49602751 ptr=EF276000 processID=00005066 xiAPI: Create handles 1 Process 00005066 xiAPI: xiReadFileFFS 'SensFPNCorrections' not found xiAPI: ScmCorrectorA::LoadAndParseConfigurationFile Warning: file SensFPNCorrections not found xiAPI: Enable sensor xiAPI: Frequency 114 71 xiAPI: Enabled 1 bandwidth 450000000 xiAPI: xiReadFileFFS 'SensFPNCorrections' not found xiAPI: ScmCorrectorA::LoadAndParseConfigurationFile Warning: file SensFPNCorrections not found xiAPI: xiReadFileFFS 'SensFPNCorrections' not found xiAPI: ScmCorrectorA::LoadAndParseConfigurationFile Warning: file SensFPNCorrections not found xiAPI: Frequency 114 71 xiAPI: Enabled 1 bandwidth 450000000 xiAPI: Time needed to read BPL:78ms xiAPI: Successfully parsed BPL file, 1229 total corrected pixels xiAPI: SetGPIO 1 set to 8 xiAPI: Frequency 110 6d xiAPI: Enabled 1 bandwidth 450000000 xiAPI: SetGPIO 1 set to 0 xiAPI: Current bandwidth limit auto-set to 3240 Mbps (min:80Mbps,max:3600Mbps) xiAPI: Frequency 110 6d xiAPI: Enabled 1 bandwidth 405000000 xiAPI: Time needed to read BPL:76ms xiAPI: Successfully parsed BPL file, 1182 total corrected pixels xiAPI: SetGPIO 1 set to 8 xiAPI: Frequency 110 6d xiAPI: Enabled 1 bandwidth 405000000 [ximea_camera_cam_fl-2] process has died [pid 20554, exit code -11, cmd /home/autonomoose/catkin_ws/devel/lib/ximea_ros_cam/ximea_ros_cam_node __name:=ximea_camera_cam_fl __log:=/home/autonomoose/.ros/log/6a1098e0-d390-11e7-a74b-0010f35f362f/ximea_camera_cam_fl-2.log]. log file: /home/autonomoose/.ros/log/6a1098e0-d390-11e7-a74b-0010f35f362f/ximea_camera_cam_fl-2*.log [ximea_camera_cam_rf-5] process has died [pid 20557, exit code -11, cmd /home/autonomoose/catkin_ws/devel/lib/ximea_ros_cam/ximea_ros_cam_node __name:=ximea_camera_cam_rf __log:=/home/autonomoose/.ros/log/6a1098e0-d390-11e7-a74b-0010f35f362f/ximea_camera_cam_rf-5.log]. log file: /home/autonomoose/.ros/log/6a1098e0-d390-11e7-a74b-0010f35f362f/ximea_camera_cam_rf-5*.log xiAPI: Frequency 110 6d xiAPI: Enabled 1 bandwidth 450000000

[jskhu]

@carloswanguw There seems to be an issue regarding the discovery of all the camera devices with multiple nodelets. It might be better to open all the devices in one process to avoid this issue and then distribute each camera instance to different nodelets.

@mpitropov I've increased the poll rate for now and added some additional measures. As mentioned before, there is an issue with camera discovery so it might take more than a few seconds for the nodelets to detect and connect with all the cameras.

[jskhu]

@mpitropov I've added a nodelet manager which fixes the camera loading issue. I've only tested with 2 cameras connected so let me know if 6 - 8 cameras causes any issues.

[mpitropov]

Okay, I'll see if I have time today to test and I'll update you with what happens.

[carloswanguw]

@jskhu I would recommend trying an indefinite polling routine (or maybe the first 20 tries or so), spaced out by 1 or two seconds for each attempt. Sort of like the situation:

Attempting to open ximea_cam_1... Failed.
Retrying in 2 seconds...
Attempting to open ximea_cam_1... Failed.
Retrying in 2 seconds...
Attempting to open ximea_cam_1... Success!

I was thinking eventually all cameras will get latched on eventually. It just takes time.

As you mentioned an alternative way would be to have a single thread to open multiple cameras, and then generate a nodelet for each camera, though I am not entirely sure how to approach this in an elegant manner.

[carloswanguw]

It takes roughly 1-2 seconds per camera on the same bus I believe, so really for 2 cameras we have 4 seconds, and for 6-8 we have 12-16 seconds.

[jskhu]

@carloswanguw I've added the camera nodelets under a common node manager which allows the XIMEA API calls to be requested serially between all the cameras. I'm not sure if this was intended, but previously, each camera was under a separate manager which caused some cameras to never connect.

Also, the cameras are indefinitely polling now in 2 second intervals. I'll add the logging as well.

[mpitropov]

It still is unable to start all cams [ximea_camera_cam_f-1] process has died [pid 18611, exit code -11, cmd /home/autonomoose/catkin_ws/devel/lib/ximea_ros_cam/ximea_ros_cam_node name:=ximea_camera_cam_f __log:=/home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_f-1.log]. log file: /home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_f-1*.log [ximea_camera_cam_lb-6] process has died [pid 18618, exit code -11, cmd /home/autonomoose/catkin_ws/devel/lib/ximea_ros_cam/ximea_ros_cam_node name:=ximea_camera_cam_lb log:=/home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_lb-6.log]. log file: /home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_lb-6*.log [ximea_camera_cam_b-8] process has died [pid 18656, exit code -11, cmd /home/autonomoose/catkin_ws/devel/lib/ximea_ros_cam/ximea_ros_cam_node __name:=ximea_camera_cam_b log:=/home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_b-8.log]. log file: /home/autonomoose/.ros/log/f5cb5334-d548-11e7-8e17-0010f35f362f/ximea_camera_cam_b-8*.log

[jskhu]

@mpitropov Can you send me your launch file? And the logs if possible.

[carloswanguw]

Closing this branch as it is not being worked on anymore by @jskhu , I am superseeding it with another internal branch: https://github.com/wavelab/ximea_ros_cam/tree/feature-10-multiple_cam_launch