search

neufieldrobotics / spinnaker_sdk_camera_driver

Point Grey (FLIR) Spinnaker based camera driver (Blackfly S etc.)
MIT License
126 stars 91 forks source link

Startup camera sync via initializing/deinitializing cycle does not work with GigE camera #10

Closed llvs closed 5 years ago

llvs commented 5 years ago

I use the spinnaker_sdk_camera_driver and I experience an issue of which I don't know what the core of the issue might be.

Your ROS driver detects the camera and the flushing also seems to work (see log of roslaunch below) However, the camera seems to not start with frame ID 0.

The camera is a new Blackfly S with standard factory firmware. It is connected via GigE with Power-over-Ethernet network cable. The camera also works fine in spinview.

The test was done in VirtualBox on a fully updated Lubuntu 16.04.

What can I do, to have the camera start with a certain frame number? Console output is as follows:

[ INFO] [1539259721.897108867]: [ OK ] USB memory: 1000 MB
20181011[ INFO] [1539259721.897183646]: *** PARAMETER SETTINGS ***
[ INFO] [1539259721.897203254]: ** Date = 20181011
[ INFO] [1539259721.897739443]:   Save path set via parameter to: /home/pw
[ INFO] [1539259721.897782787]:   Camera IDs:
[ INFO] [1539259721.898293787]:     <serial no>
[ INFO] [1539259721.898791570]:   Camera Aliases:
[ INFO] [1539259721.898814759]:     <serial no> >> cam0
[ INFO] [1539259721.899740583]:   Unique time stamps for each camera: 0
[ INFO] [1539259721.900238189]:   color set to: 0
[ INFO] [1539259721.900932690]:   Exporting images to ROS: 1
[ INFO] [1539259721.901459004]:   Showing live images setting: 0
[ INFO] [1539259721.901925390]:   Showing grid-style live images setting: 0
[ INFO] [1539259721.902395142]:   Max Rate Save Mode: 0
[ INFO] [1539259721.902891314]:   Displaying timing details: 0
[ INFO] [1539259721.904583561]:   No. of images to skip set to: 20
[ INFO] [1539259721.906793675]:   Init sleep delays set to : 20 sec
[ WARN] [1539259721.907340806]:   'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1539259721.907794313]:   'exp'=0, Setting autoexposure
[ INFO] [1539259721.908270332]:   Binning set to: 2
[ INFO] [1539259721.908731872]:   Using Software rate control, rate set to: 20
[ INFO] [1539259721.909187848]:   Saving images set to: 0
[ INFO] [1539259721.910053749]:   Camera coeffs not provided correctly, camera info messges will not be published.
[ INFO] [1539259721.910122159]: Creating system instance...
[ INFO] [1539259721.910468338]: Retreiving list of cameras...
[ INFO] [1539259723.542994586]: Numer of cameras found: 1
[ INFO] [1539259723.543034197]:  Cameras connected: 1
[ INFO] [1539259723.543082408]:   -<serial no>
[ INFO] [1539259723.545509510]: *** FLUSH SEQUENCE ***
[ INFO] [1539259723.545538521]: Initializing cameras...
[ INFO] [1539259724.172084859]: Deinitializing cameras...
[ INFO] [1539259725.952781566]: All cameras deinitialized.
[ INFO] [1539259725.952871993]: Initializing cameras...
[ INFO] [1539259726.425770815]: *** ACQUISITION ***
[FATAL] [1539259726.495451653]: ASSERTION FAILED
    file = /home/pw/ros/ws/spinnaker_ws/src/spinnaker_camera_driver/src/camera.cpp
    line = 59
    cond = frameID_ == 0
    message = [FATAL] [1539259726.495545610]: First frame ID was not zero! Might cause sync issues later...
[FATAL] [1539259726.495617142]:

You suggested in private conversation:

We dont have any GigE versions at our lab, so haven't tested with them. The assertion for the frame number is because when using multiple camera we want to make sure they are synced. If you are only using a single camera, you should be able to safely comment out the assertion at: Link to source

Your suggestion above is not possible for us, as we want to use a stereo camera pair and want to have synced cameras.

karanchawla commented 5 years ago

Having the same issue, I commented out the assertion line, however, that leads to the node crashing eventually.

ghost commented 5 years ago

What is ur setup exactly? GIG E or ethernet. How many cameras?

ghost commented 5 years ago

@karanchawla the above question is for you.

karanchawla commented 5 years ago

I’m away from the computer at the moment, will send a complete bug report in a bit.

llvs commented 5 years ago

I asked the manufacturer FLIR about this and they told me (translated quote):

There actually is a difference between the GigE and USB3 variant: The GigE camera resets FrameID, if AcquisitionStop, AcquisitionStart is executed. The USB3 does this only on EndAcquisition, BeginAcquisition. The counter cannot be reset manually.

HTH

ghost commented 5 years ago

@llvs and @karanchawla are you guys using the GPIO cable to trigger the cameras as described in the ReadMe. If you are setting up the code for 2 camera and the trigger doesn't happen correctly, the code would timeout during the image acquisition on the slave camera and terminate.

@llvs before worrying too much about the frame id's it would be worth checking if it is an issue. If you are able to acquire images from 2 cameras, I suggest pointing them at a stopwatch and checking if they are synced.

llvs commented 5 years ago

No, as the problems using the node happen already with a single camera. However, in the future I assume that we will need to employ hardware triggering via GPIO.

karanchawla commented 5 years ago

My setup is as follows: Camera: Blackfly S Sensor: RGB Connection: GIG E

I'm trying to use this repository to acquire images however, the node would crash after a few minutes after giving a few warnings about image might be incomplete!

ghost commented 5 years ago

@karanchawla which particular model are you using. How many megapixels camera is it?

karanchawla commented 5 years ago

Specs for the camera I'm using: https://www.ptgrey.com/blackfly-s-color-32-mp-gige-vision-sony-imx265

ghost commented 5 years ago

So does it work fine for a few mins?

karanchawla commented 5 years ago 
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ThinkPad-L380-Yoga:43951/

SUMMARY
========

PARAMETERS
 * /acquisition_node/binning: 2
 * /acquisition_node/cam_aliases: ['cam0']
 * /acquisition_node/cam_ids: [18434273]
 * /acquisition_node/color: True
 * /acquisition_node/delay: 0
 * /acquisition_node/exp: 0
 * /acquisition_node/frames: 0
 * /acquisition_node/live: False
 * /acquisition_node/live_grid: False
 * /acquisition_node/master_cam: 18434273
 * /acquisition_node/max_rate_save: False
 * /acquisition_node/save: False
 * /acquisition_node/save_path: ~
 * /acquisition_node/save_type: bmp
 * /acquisition_node/skip: 30
 * /acquisition_node/soft_framerate: 15
 * /acquisition_node/time: False
 * /acquisition_node/to_ros: True
 * /acquisition_node/utstamps: False
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    acquisition_node (spinnaker_sdk_camera_driver/acquisition_node)

ROS_MASTER_URI=http://localhost:11311

process[acquisition_node-1]: started with pid [4337]
[ INFO] [1539896282.597559438]: [ OK ] USB memory: 1000 MB
20181018[ INFO] [1539896282.597629787]: *** PARAMETER SETTINGS ***
[ INFO] [1539896282.597642561]: ** Date = 20181018
[ INFO] [1539896282.598052658]:   Save path set via parameter to: /home/skyryse
[ INFO] [1539896282.598085649]:   Camera IDs:
[ INFO] [1539896282.598526005]:     18434273
[ INFO] [1539896282.599001187]:   Camera Aliases:
[ INFO] [1539896282.599019405]:     18434273 >> cam0
[ INFO] [1539896282.599881492]:   Unique time stamps for each camera: 0
[ INFO] [1539896282.600393175]:   color set to: 1
[ INFO] [1539896282.600880416]:   Exporting images to ROS: 1
[ INFO] [1539896282.601281588]:   Showing live images setting: 0
[ INFO] [1539896282.601648232]:   Showing grid-style live images setting: 0
[ INFO] [1539896282.602022591]:   Max Rate Save Mode: 0
[ INFO] [1539896282.602444702]:   Displaying timing details: 0
[ INFO] [1539896282.602865232]:   No. of images to skip set to: 30
[ INFO] [1539896282.603289117]:   Init sleep delays set to : 30 sec
[ WARN] [1539896282.603729684]:   'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1539896282.604159206]:   'exp'=0, Setting autoexposure
[ INFO] [1539896282.604585260]:   Binning set to: 2
[ INFO] [1539896282.604973468]:   Using Software rate control, rate set to: 15
[ INFO] [1539896282.605421087]:   Saving images set to: 0
[ INFO] [1539896282.606245677]:   Camera coeffs not provided correctly, camera info messages will not be published.
[ INFO] [1539896282.606282651]: Creating system instance...
[ INFO] [1539896282.606560907]: Retrieving the list of cameras...
[ INFO] [1539896284.732364455]: Numer of cameras found: 1
[ INFO] [1539896284.732453493]:  Cameras connected: 1
[ INFO] [1539896284.732554350]:   -18434273
[ INFO] [1539896284.739698339]: *** FLUSH SEQUENCE ***
[ INFO] [1539896284.739773712]: Initializing cameras...
[ INFO] [1539896285.389000848]: Deinitializing cameras...
[ INFO] [1539896287.166753612]: All cameras deinitialized.
[ INFO] [1539896287.166924233]: Initializing cameras...
[ INFO] [1539896287.645791210]: *** ACQUISITION ***
[ WARN] [1539896289.066276953]: Image incomplete with image status 9!
[ WARN] [1539896290.266493358]: Image incomplete with image status 9!
.
.
.
[ WARN] [1539896340.474110609]: Image incomplete with image status 9!
[FATAL] [1539896342.525220597]: Some exception occured. 
                                                         Exiting gracefully, 
  possible reason could be Camera Disconnection...
[ INFO] [1539896342.599320622]: Deinitializing cameras...
[ INFO] [1539896343.624576840]: All cameras deinitialized.
[ INFO] [1539896343.624668475]: Clearing camList...
[ INFO] [1539896343.624731904]: Releasing camera pointers...
[ INFO] [1539896343.624787122]: Releasing system instance...
terminate called after throwing an instance of 'std::bad_cast'
  what():  std::bad_cast
karanchawla commented 5 years ago

It works for less than a minute after which it crashes. I have posted the log in the comment above.

ghost commented 5 years ago

@Karanchawla That issue seems to be with the Spinnaker SDK as opposed to the ROS driver. See https://www.ptgrey.com/KB/11142 at the end under Troubleshooting.

ghost commented 5 years ago

@karanchawla and @llvs were you guys able to get the cameras to work? Do you have an update about the issues?

mikedef commented 5 years ago

Hi @shahvi I am getting the exact same issue as @karanchawla. I have a Blackfly BFLY-PGE-20E4C. I had a frameID error to start with as above. I then commented line 59 in camera.cpp. Once I did that the camera would run for under a minute and then would crash again. Same failure as above.

I can run spinview at any framerate available. I have set frame:=10 for the ROS node.

ghost commented 5 years ago

@mikedef Its difficult to debug without us having that type of camera. One thing you could try would be to launch the code with the debug turned on by changing the first line in https://github.com/neufieldrobotics/spinnaker_sdk_camera_driver/blob/fd3d35e43ad590c4ef42937083317c56d5459386/launch/acquisition.launch#L3 to debug_console.conf

See if the debug messages help...

mikedef commented 5 years ago

@shahvi changing to debug_console.conf did not give me any updated debug messages.

Here is my latest error message: Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event [-1011] I see see that there was a closed ticket for this error, but no solution to it. Do you have any updated information on this buffer data issue? Thanks

vik748 commented 5 years ago

@mikedef , I am really out of ideas. It seems to be happening in the Spinnaker_SDK and I am not sure what it might be. Hard to know without having one to play with.

mikedef commented 5 years ago

@shahvi Right I am not sure either. Also the color flag does not work for this camera as well. Spinview works fine. So I am not sure why it's not working with this ROS app. Thanks for the help either way.

mikedef commented 5 years ago

Hi @shahvi looks like the Event Data Spinnaker error was solved by increasing UDP buffer size to 25MB by 'sudo sysctl -w net.core.rmem_max=26214400' and 'sudo sysctl -w net.core.rmem_default=26214400'. I found this on a different forum. Sorry about the back and forth. I figured since the camera worked fine with Spinview that it would work fine with the ROS node.

vik748 commented 5 years ago

@mikedef thanks for the update. So does the driver work properly once you make this change?

mikedef commented 5 years ago

@shahvi The driver does not work in color mode. It does work when in black and white. I had to make a few changes, mostly commenting out of error checking code as highlighted above.

I get the following errors when setting the color flag to true [FATAL] [1549565095.362445492]: Unable to set PixelFormat to BGR8 (entry retrieval). Aborting... [FATAL] [1549565095.362619558]: Error: Spinnaker: LogicalErrorException NULL pointer dereferenced [-2005]

vik748 commented 5 years ago

Setting the eumeration takes place in: https://github.com/neufieldrobotics/spinnaker_sdk_camera_driver/blob/fd3d35e43ad590c4ef42937083317c56d5459386/src/camera.cpp#L134

Maybe these GigE cameras have a different enumeration entry ? You can check in SpinView, what enumerations the PixelFormat option accepts.

image

mikedef commented 5 years ago

@shahvi Looks like my only option is BayerGB8. Where is this set exactly? I changed line 96 to PixelFormat_BayerGB8 and did a catkin_make, but I still get the BGR8 error as before.

vik748 commented 5 years ago

You should be able to change it at https://github.com/neufieldrobotics/spinnaker_sdk_camera_driver/blob/fd3d35e43ad590c4ef42937083317c56d5459386/src/capture.cpp#L508

mikedef commented 5 years ago

@shahvi Sorry it's actually BayerBG8 not GB. Either way it works when I change the pixel format to BayerBG8. Thanks for the help. The ROS node is now running with the default settings and the color flag set to true.

llvs commented 5 years ago

In the meantime we found the following regarding the errors with:

Image incomplete with image status 9!

The two cameras are flooding the IP connection with packets. To have more than one camera coexist in peace one has to divide the GigE bandwith between the cameras. This is done via the PacketDelay parameter which may also be called GevSCPD (Gigevision Stream Channel Packet Delay) or the corresponding parameter DeviceLinkThroughputLimit (which is easier to understand and set a meaningful value). You can set the DeviceLinkThroughputLimit to 61644209 (equals PacketDelay of 73760 for FLIR camera tick frequency) and this equals half of GigE bandwidth. Or you may use 40841495 (PacketDelay 147880) which is a third of GigE bandwidth to have some space. Please note that limiting device bandwidth of course reduces the frame rate such that the necessary data transfer does not exceed the limit.

angegu commented 5 years ago

@llvs That solves the image incomplete and event buffer timeout issue thanks!

alankwok97 commented 5 years ago

@llvs how would you set that up with ros in terminal exactly? I can find the parameters in spinview but changing in spinview doesn't seem to help in roslaunching the application as the parameters are reset the next time I open spinview.

Many thanks.

kaalessi commented 5 years ago

@alankwok97 You can save SpinView settings to your camera by typing user into the search box in SpinView. Change the parameter UserSetSelector to UserSet0 or UserSet1. Then click the execute button for UserSetSave. Now your settings are saved and all you have to do now is change UserSetDefault to match your desired UserSet. Everytime your camera is booted up, the settings saved will be used. If you want to go back to Default, you can select Default under UserSetSelector and click Execute for UserSetLoad.

alankwok97 commented 5 years ago

@kaalessi Thanks for your reply!

atyshka commented 5 years ago

@llvs what could be causing this error if I’m using only one camera? You said in your case with multiple cameras they were flooding the network, but this shouldn’t happen with a single camera, right?

llvs commented 4 years ago

I can only guess here, but it may have to do with too small network buffers. The linux defaults are maybe too little. Or you may want to use Ethernet Jumbo Frames if the camera allows it to reduce network load. But those are only guesses. HTH