ViO cumulative drift on the Z axis

[patrickpoirier51]

Hello,

I am currently experimenting outdoor with the ViO and OakD. My test setup is with the camera attached to a 5M leash and I am walkin in a circle I changed the XY scale to 20M and kept the Z scale to 2M

Here is the result graph

XY seems to be OK

Z shows a cumulative drift

Please note that I updated formware on IMU to latest ~/depthai-python/examples/IMU$ python3 imu_version.py IMU type: BNO086, firmware version: 3.9.9

spectacularAI 1.21.0 depthai 2.22.0.0

[Bercon]

We just released SDK version 1.22.0 yesterday which fixes an issue with depth map being incorrectly aligned with OAK-D cameras. This fix should also improve tracking performance. You can also enable slam, this should fix any accumulated drift when returning to previously seen area:

config = spectacularAI.depthai.Configuration()
config.useSlam = True
vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config)
...

The drift seems unsualy large and if the things listed above don't fix it, you could do a recording of it using vio_record.py script and send it to us via email for investigation

[patrickpoirier51]

Hello and thanks for you excellent support.

Tested both Oak and Realsense D455 with the new release D455 is OK now (it was drifting before) but OAK is still drifting. Configuring pipeline for SLAM makes no change

Trying to record but end up with ffmpeg error

python3 vio_record.py --output test /home/msi/sdk-examples/python/oak/vio_visu.py:103: UserWarning: frames=None which we can infer the length of, did not pass an explicit save_count and passed cache_frame_data=True. To avoid a possibly unbounded cache, frame data caching has been disabled. To suppress this warning either pass cache_frame_data=False or save_count=MAX_FRAMES. anim = FuncAnimation(fig, update_graph, interval=15, blit=True) Recording!

Close the visualization window to stop recording ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1) configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100 libpostproc 55. 5.100 / 55. 5.100 [hevc @ 0x556be1f33800] Stream #0: not enough frames to estimate rate; consider increasing probesize Input #0, hevc, from 'test/rgb_video.h265': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv, bt470bg), 1920x1080 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 1200k tbn, 30 tbc Output #0, mp4, to 'test/rgb_video.mp4': Metadata: encoder : Lavf58.29.100 Stream #0:0: Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt470bg), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 30 fps, 30 tbr, 1200k tbn, 1200k tbc Stream mapping: Stream #0:0 -> #0:0 (copy) Press [q] to stop, [?] for help [mp4 @ 0x556be1f3bbc0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 0x556be1f3bbc0] pts has no value Last message repeated 395 times frame= 396 fps=0.0 q=-1.0 Lsize= 13685kB time=00:00:13.06 bitrate=8579.9kbits/s speed= 485x
video:13680kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.040591%

[patrickpoirier51]

data.zip Here is the data.jsonl file for a very short test, if you need more I may replicate the 5M circle

[Bercon]

I don't see anything wrong in the data.jsonl, but a proper analysis would also require the video. The SDK uses the system ffmpeg i.e. the ffmpeg you get when typing ffmpeg on command line. You could try updating/switching version of that. You can also provide the video codec used for encoding via vio_python.py flag --ffmpeg_codec. The codec should support -pix_fmt yuv420p

Most of the ffmpeg log above is from the rgb data stream that doesn't pass through the SDK

[patrickpoirier51]

Hello and thanks again

Do we need additional codec to process the stream ? We can see h265 == Input #0, hevc, from 'test/rgb_video.h265': I dont see h265 when ffmpeg --decoders or -encoders , tried to compile with library but no success

Could you tell me your ffmpeg version and if h265 is part of the decoders ?

[Bercon]

The RGB video is encoded on the OAK-D device. The Python script just converts the raw h265 stream into proper video file. It's not required for tracking in default mode, so you can just disable it with --no_rgb flag

[patrickpoirier51]

OK, so i launch python3 vio_record.py --no_rgb --output test with no error

The short test has generated these files ls -l total 116156 -rw-rw-r-- 1 msi msi 943 Sep 3 11:38 calibration.json -rw-rw-r-- 1 msi msi 1947511 Sep 3 11:38 data2.mkv -rw-rw-r-- 1 msi msi 1938605 Sep 3 11:38 data3.mkv -rw-rw-r-- 1 msi msi 10542001 Sep 3 11:38 data.jsonl -rw-rw-r-- 1 msi msi 104495016 Sep 3 11:38 data.mkv -rw-rw-r-- 1 msi msi 430 Sep 3 11:38 vio_config.yaml

If this is OK , I will proceed to full test and upload the zip file Do you have any preferred method or a share google file is OK ^

[Bercon]

That looks good. You can also check the video, data.mkv should be the 16bit depth data that can be hard to playback. data2.mkv and data3.mkv are the grayscale cameras, but only include every 6th frame. Sharing through google drive is fine. You can send the link to us via email to avoid sharing it with whole internet :)

[patrickpoirier51]

Hello again,

Here is the link, https://drive.google.com/file/d/1-fXpQSJtkJEatCLpssuGNz-IYHQJQE88/view?usp=sharing I will delete once you have downloaded

We can see loop closure and adjustment on the second loop

But there is drift on long runs , I did a long run test with the Realsense D455 and there is much less drift Here is the original 5M loop and an additionnal 8 pattern with the Realsense

[Bercon]

Thank you, we've downloaded it and will investigate

[Bercon]

There aren't any obvious issues in the data, so it's difficult to pinpoint the issue. The depth map might not be very accurate here and decreasing the amount of trust the algorithm puts in it, seems to help. You can modify the depthErrorScale: 0.1 parameter in the vio_config.yaml file to depthErrorScale: 1 when replaying the session you sent us or set it as internal configuration parameter in live session. You could also try recalibrating the camera using these instructions: https://docs.luxonis.com/en/latest/pages/calibration/

[patrickpoirier51]

OK Thanks

I will work with these parameters and report back

[patrickpoirier51]

Hello

Just received OAK-D-W Proceeded with test (vio_visu.py) using release 1.22 Still have drift on the Z axis

[Bercon]

The axis scales in the picture aren't equal, so if I read this correctly there is <1m drift in z-axis over the traveled ~50m distance? 1m / 50m = 2% drift is roughly in the expected ballpark. If the return position is exactly the same, the XY drift seems smaller <0.5m perhaps, so likely the Z-drift could also be improved. If you enable SLAM, it should reduce this. I would also try setting depthErrorScale: 1 and test if that helps here as well

[patrickpoirier51]

Hello Jerry,

You are right, Z scale is not the same but this is true for the realsense D455 test as well that does not show such drift. So I will enable SLAM and set depthErrorScale: 1 and report back

[patrickpoirier51]

Hello with 1.22 pip list |grep specta spectacularAI 1.22.0

and this config pipeline = depthai.Pipeline() config = spectacularAI.depthai.Configuration() config.useSlam = True config.depthErrorScale: 1.0 vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config)

I can get some loop closure but on a more complex map (dual loop) it it drifting

[Bercon]

The current SLAM is intended for fairly small spaces and when traveling larger distances like >50m, it will forget the older parts of the map and won't be able to make loop closures

Just to confirm, the depthErrorScale should be applied as an internal parameter, like so:

pipeline = depthai.Pipeline()
config = spectacularAI.depthai.Configuration()
config.useSlam = True
config.internalParameters = { 'depthErrorScale': '1.0' }
vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config)

[patrickpoirier51]

OK thanks, noted. I will apply the modified internal parameter and report if any enhancement

Thanks again for your excellent support

[patrickpoirier51]

With the correct parameter definition it seems to work better ;-)

So I will continue pushing the test and tweak the parameters Regards