Using other datasets (Prophesee event camera PPS3MVCD)

[grammers]

I have been trying to use the EMVS on data from a Prophesee event camera PPS3MVCD without success. I have adjusted the code to run on the Prophesee_event_msgs and the topics for our motion capture system. Moreover, looking into the Prophesee ROS msgs and Davis ROS msgs seems to have a similar structure.

The code executes without errors but the output is not the expected. The first observed place with unexpected results is in the Disparity Space Image (DSI). It looks like the event rays are parallel and therefore don't intersect, causing the DSI to lack dense areas.

In the figure above, the right part of the scene should be close to the camera and the left part of the scene should be further back. Changing settings can change how clear the contours are but there are no settings that improve the depth image.

I recorded a dataset that includes geometry_msgs/TransformStamped data from Vicon Motion capture system as well as prophesee_event_msgs/EventArray data from the Prophesee camera.

The event camera is calibrated using the Prophesee camera calibration method: metavision_mono_calibration (https://docs.prophesee.ai/metavision_sdk/modules/calibration/guides/intrinsics.html?highlight=calibration)

Could you please let me know whether there are some constraints on how the datasets need to be recorded? Or do you have any insight into what could be the issue?

Thanks for your time in advance!

[guillermogb]

Hi. I assume you tried the examples in the Readme and they work fine. The question, when the rays are so parallel in the DSI, is... do you have enough (temporal) baseline? If the camera moves only a little one cannot expect to get accurate depth with EMVS.

[grammers]

Yes, the README examples work fine. I will record a new dataset where I make sure I have enough camera movement.

[grammers]

I recorded new data sets with different higher speed movements (of the camera). I played with the configurations, changed the time span, and so on. The best result I manage to get where this (one of the most promising results yet).

I don't know what else to test, any help would be appreciated. Below are links necessary to replicate what I have done.

The adapted code can be found in this repository grammers/rpg_emvs/tree/prophesee in branch prophesee. There can the .conf I used also be found. The bag file can be found here.

[guillermogb]

The topic /prophesee/camera/camera_info is empty. Do you have the intrinsic calibration of the camera that you used to record the sequence?

[grammers]

Yes, I have hardcoded the camera calibration in main.cpp

[guillermogb]

My guess is that the data is not good... Sorry if it sounds like "blaming others". I inspected the data; see the results and video attached here, recorded at half the speed. The first 1.8 s of data or so look ok, but there is not enough baseline to produce a good 3D reconstruction (results and conf file in the link above). After that, the events look like coming from "washed out" edges instead of sharp ones (I do not know why - maybe there is some bottleneck in the data transmission or acquisition) thus, it appears to me that they are noisy and EMVS cannot do much with them.

Suggestions:

• always visualize the data first, to see if it looks of "good quality". Adjust camera and scene parameters for optimal acquisition.
• conf file: set dimX and dimY to the camera resolution (rather than arbitrary 150x150)
• conf file: set the FOV to approximately the FOV of the camera (rather than 110).
• conf file: when testing, try to use about 1-2 seconds of data on the same DSI.
• need enough (temporal) baseline (in this 1-2 second interval used for the DSI).
• motion: if the camera moves parallel to the scene edges, then few events will be generated, so do not expect to reliably reconstruct those edges. Try testing with a circular motion ("wax on") parallel to the image plane, to obtain events from edges in all directions.

Cheers

[grammers]

I'm having problems finding a good camera bias tuning, to achieve good data. If you wid you're experience can give me any suggestions of how to find them would I appreciate it? The parameters I have are the following.

Name	My value in mV	Description
bias_pr	1250	impacts the operating point of the event-based pixel front end, mainly the photodiode
bias_fo	1580	controls the pixel low-pass cut-off frequency
bias_hpf	1500	controls the pixel high-pass cut-off frequency
bias_refr	1800	controls the refractory period. It represents a dead time for which the pixel will be kept in rest after an event acknowledgment.
bias_diff	300	controls the reference voltage of the comparators
bias_diff_on	460	controls the thresholds for positive events
bias_diff_off	220	controls the thresholds for negative events

I also have an "activity filter temporal depth" that can be tuned, it is currently set to 2ms.

[guillermogb]

Maybe this is a question for the support team at Prophesee, sorry.