Enabling camera capture for VIO applications

[Jaeyoung-Lim]

Describe problem solved by the proposed feature

Pixahakws already have all the necessary GPIOs to do camera synchronization with IMUs for VIO applications.

However, it seems that the current camera capture system is built for a specific use case of mapping with PPK, which is making the following assumptions

• Camera images are always triggered through a trigger signal. This assumption excludes the use of free-running cameras
• GPS is always available when taking images.

The following image shows a free-running camera connected to the FMU, where only the capture signal is connected. The camera_capture driver correctly captures the images when the camera is running.

However, the capture signals are never logged since there are no trigger signals.

Time synchronization of trigger signals is not limited to video, but also lidar/radar sensors. Having a general framework for enabling such sensors integrated to the fmu will help integrate these sensor applications with the fmu.

@Igor-Misic FYI

Additional things I found

• The camera_capture driver publishes a camera_trigger uORB message when a capture signal is received (https://github.com/PX4/PX4-Autopilot/blob/95b30056794b47bb415f4c1d96028ec77a567446/src/drivers/camera_capture/camera_capture.cpp#L222)
• The camera_trigger driver uses the camera_trigger uORB message to trigger the camera (https://github.com/PX4/PX4-Autopilot/blob/95b30056794b47bb415f4c1d96028ec77a567446/src/drivers/camera_trigger/camera_trigger.cpp#L855)

Describe your preferred solution

A way of integrating a capture signal that is not necessarily triggered by a trigger signal, and still be able to later synchronize camera images with imu time stamps offboard

One of the use cases would be using a free running camera (e.g. UVC) and be able to synchronize imus with the images. This would enable visual-inertial sensor data that can be utilized for advanced applications such as VIO(Visual Inertial Odometry).

• camera_trigger and camera_capture messages should be used to capture different information, not sequentially
• camera_feedback module should log camera capture information without the dependency of state estimates (gps, attitude)

Describe possible alternatives

• We can try to generalize the capture interface all together to be used outside for any capture signal available

Additional context

• This problem was briefly discussed in the dev call last week

[dagar]

I think the main problem here is that we start with fairly general simple implementations of things that are justifiably named broadly/generically (eg "camera_trigger", "camera_feedback", etc), but then over time things evolve to do a better job solving some specific problem for a product or use case.

From my perspective the right solution for us is to lean on the modularity of the system, and wherever possible get out of each others way. I would like to have more dedicated modules solving one specific problem, and when someone inevitably has a kinda/sorta similar problem to solve we create a new module.

So what I would propose is that we create a new dedicated capture module for your specific use case and rename the existing modules. We need to minimize all this coupling in the system, there are too many areas where it becomes very difficult to move forward because no one person is in a position to test or even understand all the different pieces and use cases.

[tiralonghipol]

@Jaeyoung-Lim @dagar hei guys, what is the status of this?

[Jaeyoung-Lim]

@tiralonghipol It is as far as written in the PR.