driver not adapted to Ladybug format 7 mode 7 (compressed images)

[bricerebsamen]

With the ladybug3, in format 7 mode 7, the images come JPEG compressed. A DC1394 frame is actually 24 images (6*RGGB), compressed, and stacked together.

I am still figuring the details out with PointGrey support, but it seems that changing the ROI does not affect which part of the image we see, but rather the compression factor. Also, I don't know if any other camera has this mode.

Anyway, the data sent from the camera is not strictly speaking an Image. And besides the semantic problem, which I would happily ignore, the main issue is that, due to compression, the number of bytes is different (usually smaller) from what can be computed from the ROI (W_H_D). Meaning that with the current driver, there are plenty of useless bytes that are published.

One idea, probably the simplest, would be to modify Format7::unpackData so that with the proper flags setup (i.e. ladybugmode), it would resize the Image.data field so that it would hold as few padding bytes as possible. We'd probably need to set a special encoding string too. Also, the other fields of the image msg would have to be ignored at the subscriber level.

Trying to use a CompressedImage instead of Image, or maybe a new data type (i.e. LadybugImageSet), seems quite difficult (more from an interface perspective than from a coding perspective). The main difficulty is to handle changing from one mode to another at runtime, which means that it would publish to a topic or another accordingly.

Any suggestion?

[jack-oquin]

I don't know much about the Ladybug, and don't have any clear idea of the best solution.

The central concept of camera1394 has always been to provide a generic ROS interface to libdc1394. I've been happy with that approach. People use it successfully with many different cameras. Yet, there is no camera-specific code in the driver, so I can maintain it using only a modest set of test cameras.

I am reluctant to add device-specific logic, because I will never be able to test it properly. While there are some code paths in the driver that none of my cameras can exercise, at least they are specified in the IIDC standard.

But, that specification itself is very loose. Format 7 in particular allows camera makers to do almost anything they want, and PGR is clearly pushing the envelope. Their Bumblebee cameras also produce rather wacky "images". I doubt PGR much cares about standards-compliance or open-source software. If no one wanted to use those devices, we would just ignore them. But, they do get a fair amount of interest in the ROS community. So, we need a solution that can be maintained for a useful period of time.

I don't personally feel qualified to do that. Maybe @chadrockey has some good ideas for how to proceed. He already mentioned the pointgrey_camera_driver in NREC/camera_drivers. Have you taken a look at that? I think we could either use it or adapt a modified version of camera1394 to specifically handle various PGR camera peculiarities.

I am willing to help write or adapt whatever code base makes the most sense. I would consider exporting an internal camera1394 C++ library API on which to build device-specific drivers, if that makes sense. But, I don't want to maintain code for devices I will probably never be able to test.

There may be more people on the ROS drivers SIG mailing list who could help us figure this out. Perhaps we should ask there.

[chadrockey]

@bricerebsamen My thoughts are that that is way too specific to be merged into camera1394. The technical reference manual states: "The implementation of these modes and the frame rates that are possible are not specified by the IIDC, and are subject to change across firmware versions"

I would try pointgrey_camera_driver, as it uses FlyCap and has a chance of working for you. You might be able to get individual camera frames with this method.

More likely though, you'll need to create your own package, 'ladybug_camera_driver' and use the ladybug SDK. This should allow you to get fully merged panoramic frames and use all of the software PointGrey has written for you. http://ww2.ptgrey.com/sdk/LadyBug

Here's some starter code they provide for grabbing panos from the LadyBug and dumping them as bmp (from here, it should be trivial to create a ROS wrapper, see pointgrey_camera_driver for an example of how I wrapped libflycap). http://www.ptgrey.com/products/ladybugSDK/examples/LadybugSimpleGrab.html

I looked up the Formats in the Ladybug Technical Reference Manual (page 32): http://www.ptgrey.com/support/downloads/downloads_admin/Dlds/Ladybug3%20Technical%20Reference.pdf

Looks like you should be able to use Format 0 and it will send the frames in order. Does your application require the high frame rate/compressed image mode?

If you're near Willow Garage, I'd be willing to take a look if you bring the camera over.

[bricerebsamen]

Thanks Jack and Chad for your feedback.

@chadrockey The ladybug sdk is attractive but it seems to be windows only. I would definitely have gone this way otherwise. I took a look at the NREC driver, but it does not support the format7 that I need (the only one able to yield color images at a high fps). So I'd have to modify it heavily, and then end up in the same situation as with the camera_1394 driver.

I guess I will fork camera_1394. I will probably make it public on github as well so that it can benefit other ladybug3 users. @jack-oquin I will get back to you at that point to discuss the licensing issues.

[chadrockey]

@bricerebsamen

Do any of the drivers support publishing the "RAW" image that you desire (the stacked, compressed images?)

If they do, the best course of action is to write a nodelet that codes this sensor_msgs/Image into sensor_msgs/CompressedImage. PointGrey clearly states this format could change between firmware revisions, and it would be easier to maintain multiple nodelets instead of multiple drivers for each firmware version (ouch!). This also avoids the issue of having to fork and keep up with camera1394 changes.

With nodelets, the overhead of useless bytes published is not as severe.

[bricerebsamen]

That's a good suggestion indeed.

However, the ladybug3 offers a bunch of control that are quite unique to it, and that would justify a fork. Namely, setting the image compression level, allowing for individual exposure control, controlling which camera image is transmitted (I will ignore the top camera), etc.

On 08/20/2013 09:56 AM, Chad Rockey wrote:

@bricerebsamen https://github.com/bricerebsamen

Do any of the drivers support publishing the "RAW" image that you desire (the stacked, compressed images?)

If they do, the best course of action is to write a nodelet that codes this sensor_msgs/Image into sensor_msgs/CompressedImage. PointGrey clearly states this format could change between firmware revisions, and it would be easier to maintain multiple nodelets instead of multiple drivers for each firmware version (ouch!). This also avoids the issue of having to fork and keep up with camera1394 changes.

With nodelets, the overhead of useless bytes is not as severe.

— Reply to this email directly or view it on GitHub https://github.com/ros-drivers/camera1394/issues/17#issuecomment-22959902.

[jack-oquin]

Brice,

Forking camera1394 is an easy path forward from where you are right now. That seems reasonable to me. In some ways, focusing on a single device makes the problem more tractable.

Something similar was done with the camera1394stereo driver, which handles bumblebee-specific issues. As far as I can tell, that project has been successful, it is released and actively maintained.

In the long run, merging bug fixes and enhancements between several different forks may become burdensome. If it does, we can work on a more modular design. It might be possible for device-specific drivers to supply derived implementations of certain base classes provided by the generic driver.That will probably require us to clean up the base implementation and figure out what kinds of modular interfaces make sense. I have no objection to doing that, but do not plan to work on it unless someone needs it.

Using git to merge changes from the base driver may prove sufficient. I am reasonably diligent about documenting the camera1394 change history, linking significant updates to the corresponding issues.