search

neufieldrobotics / spinnaker_sdk_camera_driver

Point Grey (FLIR) Spinnaker based camera driver (Blackfly S etc.)
MIT License
126 stars 91 forks source link

Multi-cam in acquisition.launch not initializing #86

Closed capaulson closed 4 years ago

capaulson commented 4 years ago

System Description Camera Name and Model: FLIR Firefly Operating System: Ubuntu 18.04 Spinnaker Version: 2.0.0.109 ROS Version: Melodic

Computer details Processor: ARM ( Nvidia Xavier ) RAM: 32GB

Multiple Cameras Only How is the triggering setup? Master/Slave or all slaves or all software triggered? All software triggered

For USB 3.0 cameras: On same USB 3.0 bus or different buses? Different

Is the USB 3.0 hub externally powered? No.

Do the cameras work with SpinView? Yes, all three running at the same time

Describe the bug Code works differently depending on how many cameras get launched.

Error Messages One camera works fine. Two or more cameras: 

[ INFO] [1589650635.976668104]: Initializing cameras...
terminate called after throwing an instance of 'Spinnaker::Exception'
  what():  Spinnaker: Camera is not started. [-1002]
ghost commented 4 years ago

First off we haven't tried a Firefly camera with our code.

I suspect it could be an issue with power. I would suggest trying a powered USB hub or powering the cameras directly from GPIO.

Also, how are you connecting them to different USB3.0 buses? Does the Xavier have more than one bus? Different USB ports doesn't necessarily mean different buses. You can check by running lsusb and looking at the bus id.

Also to rule out USB bandwidth related issues, I'd suggest trying greyscale, low framerate and with binning (to reduce resolution).

capaulson commented 4 years ago

Thanks for the debugging tips. All the USBs are on the same bus. In SpinView, I can get 24 fps per camera with all three running.

The cameras work in Spinview just off bus power, but I've also tested with a powered hub. I plan on hooking up power at the GPIO just to verify that power isn't the issue.

I tried running at 1 fps and 2 binning with the ROS driver, which failed with the same result posted above. I have hardware on order to get them on different USB buses, and I'll verify power to each camera. I'll follow up once thats done. Again, thanks for the prompt reply.

ghost commented 4 years ago

We have managed to use 3 cameras on the same bus as long as the bandwidth is accounted for, however we have always had to use external power even with a laptop.

I have never seen this Camera not started exception. From the documentation it looks like it is thrown when

If EndAcquisition() is called without a prior call to BeginAcquisition() an error message "Camera is not started" will be thrown. All Images that were acquired using GetNextImage() need to be released first using image->Release() before calling EndAcquisition(). All buffers in the input pool and output queue will be discarded when EndAcquisition() is called.

Could be that the Firefly behaves a little differently. Can you try to figure out exactly where in the code it is throwing this error.

capaulson commented 4 years ago

So I've excluded camera power as the cause of this. I did find one indication of an error in system logs: traps: nodelet[22782] trap int3 ip:7f04b82554e2 sp:7fff779d7260 error:0 in libacquilib.so[7f04b8181000+162000]

Not sure if that helps, so I'll keep digging into the code to see if I can find the issue. I've also verified this on a non-ARM processor as well, and the behavior is identical. The odd thing is that one camera works fine, every time.

capaulson commented 4 years ago

So still working on confirming where the problem is occurring. With three cameras attached to the system, I can successfully run the FLIR provided AcquisitionMultipleCamera_C. I run the ROS driver, and it fails as noted above. I then rerun AcquisitionMultipleCamera_C and it fails, claiming it can't get the next frame. I don't understand much yet about how this driver code works, @shahvi can you think of anything that get's configured that might be causing this behavior?

ghost commented 4 years ago

Can you post the full output from running the driver. Also turn on debugging by editing this line - https://github.com/neufieldrobotics/spinnaker_sdk_camera_driver/blob/2f9f14df0dcff7b4d5e1559abad8715f68d02170/launch/acquisition.launch#L3 and changingstd_console.conf to debug_console.conf

ghost commented 4 years ago

also have you setup hardware triggering as shown in the readme ?

ghost commented 4 years ago

Sorry just saw your description again, our code is setup for a master / slave arrangement only. That is the only way we can ensure that the images are taken at the same time. If you want to software trigger all of them you will definitely need some modification to the code. This might be why the spinnaker example doesn't work after running this driver. However, the error you mention is not what we would see when trying to use hardware triggering without having the cable hooked up like that.

capaulson commented 4 years ago

After wiring these up, it worked as expected. Closing. Thanks for the help!