Sequence number Drifts

[codesian]

Opening a depth disparity and a RGB 4K stream, makes that frame sequence numbers drift over time. Lowering FPS setting seems to not affect in sync, higher frame rates increase drift. Depth settings and RGB align seems to not affect, behavior its same.

C++ example

This is information from my board:

[14442C1011785ED700] [1.2.3] [3.354] [system] [info] PRINT:LeonCss: bss_init took 40.531 ms
BootloaderConfig.options1 checksum doesn't match. Is: 0x102C4414 should be: 0x89ABBD81
GPIO boot mode 0x16, interface USBD
Setting aons(0..4) back to boot from flash (offset = 0)
====ENABLE WATCHDOG====1
initial keepalive, countdown: 10
PLL0: 700000 AUX_IO0: 24000 AUX_IO1: 24000 MCFG: 24000 MECFG: 24000
Board init ret 3
eeprom configuration version: 55AA0006
Reading VERSION 6 --- ->
eeprom configuration load from user area, status: 0
--> brdInit ...
brdInitAuxDevices: Error: SC = 27: io_initialize expander_cam_gpios_1 [OK]

spi_N25Q_init: Flash JEDEC ID: ff ff ff
Invalid Flash JEDEC ID... No NOR available
Could not register the spi device
Failed to probe IR driver LM3644
Failed to probe IR driver LM3644
Failed to probe IR driver LM3644
Closing EEPROm!
MyriaX board configuration
pll0 frequency: 700000, ref0 frequency: 24000
Is booted from flash by bootloader: 0
Networking not available...
Called by: LOS, controller: LOS
Enumerating on socket: Cam_A / RGB / Center
Probe failed I2C1 0x1a reg 0016: expected 0214, read 0477
Probe failed I2C1 0x1a reg 0016: expected 0214, read 0477
Probe failed I2C1 0x1a reg 0016: expected 0378, read 0477
osDrvGt24c64Init: No EEPROM found. Will operate without
  >> Registered camera A12N02A (imx378) as /dev/Camera_0
Enumerating on socket: Cam_B / Left
  >> Registered camera TG161B (ov9282) as /dev/Camera_1
Enumerating on socket: Cam_C / Right
  >> Registered camera TG161B (ov9282) as /dev/Camera_2
Enumerating on socket: CAM_D
initial keepalive, countdown: 9
Initializing XLink...
UsbPumpVscAppI_Event: 5 VSC2_EVENT_ATTACH
UsbPumpVscAppI_Event: 4 VSC2_EVENT_RESET
initial keepalive, countdown: 8
UsbPumpVscAppI_Event: 4 VSC2_EVENT_RESET
UsbPumpVscAppI_Event: 0 VSC2_EVENT_INTERFACE_UP
Done!
Usb connection speed: Super - USB 3.0
I: [Timesync] [   1807825] [main] startSync:116     Timesync | Callback not set

Discussion at discord

[codesian]

Sequence numbers are a total disaster, as you can see, drift over time, we let the cameras start, at 30fps to show frame sequence, source of image and timestamp, for example:

SEQ	SOURCE	TIMESTAMP
14800	COLOR	495108749637
14747	DISPARITY	494817001587
14802	COLOR	495175183360
14749	DISPARITY	494883632532
14804	COLOR	495241616975
14752	DISPARITY	494983573518
14806	COLOR	495308050837
14754	DISPARITY	495050203940
14809	COLOR	495407702637
14756	DISPARITY	495116839211	(COLOR: 14800 - DISP:14756) (495108749637 - 495116839211)/1M = 8,089574 ms [Sequence difference 44 frames]
14811	COLOR	495474135457
14759	DISPARITY	495216777394	(COLOR: 14804 - DISP:14759) (495241616975 - 495216777394)/1M = -24,839581 ms [Sequence difference 45 frames]
14814	COLOR	495573786738
14761	DISPARITY	495283407922	(COLOR: 14806 - DISP:14761) (495308050837 - 495283407922)/1M = -24,642915 ms [Sequence difference 45 frames]

As far as i can test nearest frame are not more than 10 frames later.

Well now, the question its.. If the images are RGB align, its based on frame sequences or its based on timestamp, what can i find? a 45 frames missaligned images. Don know what happend.

[themarpe]

CC: @szabi-luxonis

@codesian seq numbers aren't actively aligned, apart from initial sync, as far as I recall - so any frame drop could throw those off. I'd suggest relying on getDeviceTimestamp for syncing the frames.

[SzabolcsGergely]

They indeed aren't, not sure why. Cc: @alex-luxonis since it's RGB depth alignment related.

[jasonm1892]

@alex-luxonis was responsible for RGB alignment. Why are you ignoring this one too? The RGB alignment you wrote was crashing (similar to Stereo) until someone else fixed it a year later. History is on github.