gen2-record-replay don't record disparity and depth streams on oak-d pro ff

[AnrzejAF]

Description

After reinstall Ubuntu 20.04 LTS i can't record depth and disparity using gen2-record-replay project

Steps to reproduce

1. Virtual env for python
2. Clone depthai-experiments
3. Install requirements.txt
4. Run record.py with argument --save depth

Error

(record_test) anrzej@anrzej:~/virtual/record_test/depthai-experiments/gen2-record-replay$ python3 record.py --save depth
Found 1 devices
Traceback (most recent call last):
  File "record.py", line 145, in <module>
    run()
  File "record.py", line 95, in run
    recording.start() # Start recording
  File "/home/anrzej/virtual/record_test/lib/python3.8/site-packages/depthai_sdk/record.py", line 132, in start
    self.pipeline, self.nodes = self._createPipeline()
  File "/home/anrzej/virtual/record_test/lib/python3.8/site-packages/depthai_sdk/record.py", line 280, in _createPipeline
    nodes['stereo'] = self.pm.createDepth(control=False, pipeline=pipeline, args = self.args)
  File "/home/anrzej/virtual/record_test/lib/python3.8/site-packages/depthai_sdk/managers/pipeline_manager.py", line 391, in createDepth
    return self.createDepth(
  File "/home/anrzej/virtual/record_test/lib/python3.8/site-packages/depthai_sdk/managers/pipeline_manager.py", line 422, in createDepth
    self.nodes.stereo.setOutputSize(*self.nodes.camRgb.getPreviewSize())
AttributeError: 'types.SimpleNamespace' object has no attribute 'camRgb'

Environment

• Ubuntu 20.04 LTS
• OAK-D PRO
• Python 3.8.10

Some observation

• Only depth and disparity don't work

[Erol444]

Hi @AnrzejAF , We apologize for this issue. I believe it would be best to use SDKs Record functionality. We have recently added full rosbag/db3 recording capability as well (PR here, should get merged this week). Thoughts? Thanks, Erik

[AnrzejAF]

Hi @Erol444 Sorry for late reply, but I have still some problems with depth recording. These are the things i did.

First attempt

I've tried to record depth to rosbag

Code:

from depthai_sdk import OakCamera, RecordType
with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080P', fps=20, encode='H265')
    left = oak.create_camera('left', resolution='800p', fps=20, encode='H265')
    right = oak.create_camera('right', resolution='800p', fps=20, encode='H265')
    stereo = oak.create_stereo(left=left, right=right)
    # Synchronize & save all (encoded) streams
    # oak.record([color.out.encoded, left.out.encoded, right.out.encoded], './colors', RecordType.VIDEO)
    oak.record([stereo.out.depth], './bag',RecordType.BAG)
    # oak.record([color.out.encoded, left, right, stereo.out.depth], './mcap', RecordType.MCAP)
    # Show color stream
    # oak.visualize([color.out.camera], scale=2/3, fps=True)
    oak.start(blocking=True)

Error message:

Closing OAK camera
Traceback (most recent call last):
  File "alter_record.py", line 11, in <module>
    oak.record([stereo.out.depth], './bag',RecordType.BAG)
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/oak_camera.py", line 428, in record
    record = Record(Path(path).resolve(), record_type)
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/record.py", line 72, in __init__
    from .recorders.rosbag_recorder import RosbagRecorder
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/recorders/rosbag_recorder.py", line 11, in <module>
    from rosbags.rosbag1 import Writer
ModuleNotFoundError: No module named 'rosbags'

Second attempt

I've tried to record depth to MCAP file

Code:

from depthai_sdk import OakCamera, RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080P', fps=20, encode='H265')
    left = oak.create_camera('left', resolution='800p', fps=20, encode='H265')
    right = oak.create_camera('right', resolution='800p', fps=20, encode='H265')
    stereo = oak.create_stereo(left=left, right=right)

    # Synchronize & save all (encoded) streams
    # oak.record([color.out.encoded, left.out.encoded, right.out.encoded], './colors', RecordType.VIDEO)
    # oak.record([stereo.out.depth], './bag',RecordType.BAG)
    oak.record([color.out.encoded, left, right, stereo.out.depth], './mcap', RecordType.MCAP)
    # Show color stream
    # oak.visualize([color.out.camera], scale=2/3, fps=True)

    oak.start(blocking=True)

Error message:

Closing OAK camera
Traceback (most recent call last):
  File "alter_record.py", line 12, in <module>
    oak.record([color.out.encoded, left, right, stereo.out.depth], './mcap', RecordType.MCAP)
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/oak_camera.py", line 428, in record
    record = Record(Path(path).resolve(), record_type)
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/record.py", line 66, in __init__
    from .recorders.mcap_recorder import McapRecorder
  File "/home/anrzej/.local/lib/python3.8/site-packages/depthai_sdk/recorders/mcap_recorder.py", line 5, in <module>
    from mcap_ros1.writer import Writer as Ros1Writer
ModuleNotFoundError: No module named 'mcap_ros1'

Have you idea how make this right? Any help will be appreciated

[ghost]

I ran into this error as well, it turns out you cannot save just the depth stream, you have to pass in both depth and color in order to get the depth bag.

However, the mcap issue is really strange and I have not been able to find a solution yet.

[Erol444]

Hi @AnrzejAF @AJsamk , I apologize for the delay. We have added support for ros1 (.bag)/ros2 (.db3) on this PR. It requires rosbags python module (pip install rosbags). MCAP recording doesn't work (yet), as it requires ROS installed to create ROS messages. Still wondering what's the best approach to take for MCAP recording. Thoughts? Thanks, Erik

[ghost]

Hi @Erol444,

Thanks for the response, I am looking to record mcap directly from the app which as you said does not work yet.

Quick note, I am of course able to record mcap with the depthai-ros package. But idk if it just my imagination but the depth stream there seems too dark (if that makes any sense) compared to the stream I am able to see in the depthai demo.

I am trying to basically just capture the raw stream output from both pipelines and ran into this neat little problem. AttributeError: 'types.SimpleNamespace' object has no attribute 'camRgb'.

And I found that you just have to save both depth and color streams since this is a similar issue with depthai-ros.

so running python3 record.py --save color depth --fps 30 -q BEST works just fine without the PR you mentioned.

However, with the PR, it seems to require python3-rospkgs as it throws a ModuleNotFoundError: No module named 'std_msgs'. Either that or I need to have both ROS2 and ROS installed which is not ideal.

[Erol444]

Hi @AJsamk , Regarding depth stream too dark - likely it's disparity stream, which by default is 0..95, and in depthai_demo it gets normalized to 0.255, so it's "brighter". Regarding the record.py - did you install requirements.txt - what's your depthai-sdk version, is it 1.2.3? Thanks, Erik

[ghost]

Hi @Erol444,

Thanks for that info! I will check that since I already have a few bags.

As for the record.py, as mentioned it works just fine. I was more referring to the PR solution, which threw the ModuleNotFoundError.

I did install requirements and I believe the version is 1.9.3.1 (latest)

A quick question, might be obvious but, is the pixel value of the raw depth stream (or disparity stream) the actual z-axis value? Or do I have to use the converted_depth image from the depth_image_proc::ConvertMetricNode for that?

[flipphillips]

Using, say, oak.record I am able to capture RecordType.ROSBAG just fine, but when I try to go RecordType.MCAP I'm greeted with -

No module named 'sensor_msgs'

If I understand, this is part of ROS, but I'm just trying to do this with a simple 3-4 liner, no ROS installed.

[RuixiaoZhang]

Using, say, oak.record I am able to capture RecordType.ROSBAG just fine, but when I try to go RecordType.MCAP I'm greeted with -

No module named 'sensor_msgs'

If I understand, this is part of ROS, but I'm just trying to do this with a simple 3-4 liner, no ROS installed.

Same. I try to use cv_bridge to extract depth bag, but i also obtained the error No module named 'sensor_msgs'

[Erol444]

Yes mcap requires ros installed, please use rosbag format as it doesn't require ros to record (uses python rosbag library). Thanks, Erik