search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.53k stars 4.81k forks source link

Misalignment when using the D435i #4050

Closed jonblob closed 5 years ago

jonblob commented 5 years ago
Required Info
Camera Model D435i
Firmware Version 05.11.06.200
Operating System & Version Win 10
Platform PC
SDK Version 2.22.0
Language Python 3
Segment Robot

Issue Description

Alignment of depth and color frame returns misaligned images (both depth to color and color to depth). I tested aligning the images using pyrealsense2.align as well as the rs-align tool, both yielding similar results where the aligned depth image seems to be shifted to the left with respect to the color image.

Align1 Figure 1: Color frame and corresponding aligned depth frame using pyrealsense2.align

rs-align Figure 2: Results of rs-align.

Source code:

import numpy as np
import cv2
import pyrealsense2 as rs

#Camera streams
res_color = (848,480)
res_depth = (848,480)
fps = 15

#Colorizer for the depth frame
colorizer = rs.colorizer()

#Initialize D435
pipe_D435 = rs.pipeline()
config_D435 = rs.config()
config_D435.enable_stream(rs.stream.depth, res_color[0], res_color[1], rs.format.z16, fps)
config_D435.enable_stream(rs.stream.color, res_depth[0], res_depth[1], rs.format.bgr8, fps)
align_to = rs.stream.color
align = rs.align(align_to)

#Start pipeline
prof_D435 = pipe_D435.start(config_D435)

while True:
    frames_D435 = pipe_D435.wait_for_frames()

    #alignment
    frames_D435 = align.process(frames_D435)

    #Get color and depth frames for display
    color_frame = frames_D435.get_color_frame()
    depth_frame = frames_D435.get_depth_frame()

    color_frame = np.asanyarray(color_frame.get_data())
    depth_frame = np.asanyarray(colorizer.colorize(depth_frame).get_data())

    camera_images = np.vstack((color_frame, depth_frame))

    #Display images
    cv2.imshow("RealSense", camera_images)            
    key = cv2.waitKey(1)
    if key & 0xFF == ord('q'): 
        cv2.destroyWindow("RealSense")
        pipe_D435.stop()  
        break
ev-mp commented 5 years ago

@jonblob hello, What is the log of rs-enumerate-devices -c ?

jonblob commented 5 years ago

The output of rs-enumerate-devices -c is

Device info:
    Name                          :     Intel RealSense D435I
    Serial Number                 :     846112070024
    Firmware Version              :     05.11.06.200
255.255.255.255
    Recommended Firmware Version  :     05.10.03.00
    Physical Port                 :     \\?\usb#vid_8086&pid_0b3a&mi_00#7&2d610992&1&0000#{24e552d7-6523-47f7-a647-d3465bf1f5ca}\global
    Debug Op Code                 :     15
    Advanced Mode                 :     YES
    Product Id                    :     0B3A
    Usb Type Descriptor           :     3.2

Stream Profiles supported by Stereo Module
 Supported modes:
    stream       resolution      fps       format
    Infrared 2    1280x800      @ 30Hz     Y8
    Infrared 1    1280x800      @ 30Hz     Y8
    Infrared 1    1280x800      @ 25Hz     Y16
    Infrared 2    1280x800      @ 25Hz     Y16
    Infrared 2    1280x800      @ 15Hz     Y16
    Infrared 1    1280x800      @ 15Hz     Y16
    Infrared 2    1280x800      @ 15Hz     Y8
    Infrared 1    1280x800      @ 15Hz     Y8
    Infrared 1    1280x720      @ 30Hz     Y8
    Infrared 2    1280x720      @ 30Hz     Y8
    Infrared 2    1280x720      @ 15Hz     Y8
    Infrared 1    1280x720      @ 15Hz     Y8
    Infrared 2    1280x720      @ 6Hz      Y8
    Infrared 1    1280x720      @ 6Hz      Y8
    Infrared 1    848x480       @ 90Hz     Y8
    Infrared 2    848x480       @ 90Hz     Y8
    Infrared 1    848x480       @ 60Hz     Y8
    Infrared 2    848x480       @ 60Hz     Y8
    Infrared 1    848x480       @ 30Hz     Y8
    Infrared 2    848x480       @ 30Hz     Y8
    Infrared 1    848x480       @ 15Hz     Y8
    Infrared 2    848x480       @ 15Hz     Y8
    Infrared 2    848x480       @ 6Hz      Y8
    Infrared 1    848x480       @ 6Hz      Y8
    Infrared 2    848x100       @ 100Hz    Y8
    Infrared 1    848x100       @ 100Hz    Y8
    Infrared 1    640x480       @ 90Hz     Y8
    Infrared 2    640x480       @ 90Hz     Y8
    Infrared 2    640x480       @ 60Hz     Y8
    Infrared 1    640x480       @ 60Hz     Y8
    Infrared 2    640x480       @ 30Hz     Y8
    Infrared 1    640x480       @ 30Hz     Y8
    Infrared 2    640x480       @ 15Hz     Y8
    Infrared 1    640x480       @ 15Hz     Y8
    Infrared 2    640x480       @ 6Hz      Y8
    Infrared 1    640x480       @ 6Hz      Y8
    Infrared 1    640x400       @ 25Hz     Y16
    Infrared 2    640x400       @ 25Hz     Y16
    Infrared 1    640x400       @ 15Hz     Y16
    Infrared 2    640x400       @ 15Hz     Y16
    Infrared 2    640x360       @ 90Hz     Y8
    Infrared 1    640x360       @ 90Hz     Y8
    Infrared 1    640x360       @ 60Hz     Y8
    Infrared 2    640x360       @ 60Hz     Y8
    Infrared 1    640x360       @ 30Hz     Y8
    Infrared 2    640x360       @ 30Hz     Y8
    Infrared 2    640x360       @ 15Hz     Y8
    Infrared 1    640x360       @ 15Hz     Y8
    Infrared 1    640x360       @ 6Hz      Y8
    Infrared 2    640x360       @ 6Hz      Y8
    Infrared 2    480x270       @ 90Hz     Y8
    Infrared 1    480x270       @ 90Hz     Y8
    Infrared 2    480x270       @ 60Hz     Y8
    Infrared 1    480x270       @ 60Hz     Y8
    Infrared 2    480x270       @ 30Hz     Y8
    Infrared 1    480x270       @ 30Hz     Y8
    Infrared 2    480x270       @ 15Hz     Y8
    Infrared 1    480x270       @ 15Hz     Y8
    Infrared 2    480x270       @ 6Hz      Y8
    Infrared 1    480x270       @ 6Hz      Y8
    Infrared 2    424x240       @ 90Hz     Y8
    Infrared 1    424x240       @ 90Hz     Y8
    Infrared 1    424x240       @ 60Hz     Y8
    Infrared 2    424x240       @ 60Hz     Y8
    Infrared 2    424x240       @ 30Hz     Y8
    Infrared 1    424x240       @ 30Hz     Y8
    Infrared 2    424x240       @ 15Hz     Y8
    Infrared 1    424x240       @ 15Hz     Y8
    Infrared 2    424x240       @ 6Hz      Y8
    Infrared 1    424x240       @ 6Hz      Y8
    Depth         1280x720      @ 30Hz     Z16
    Depth         1280x720      @ 15Hz     Z16
    Depth         1280x720      @ 6Hz      Z16
    Depth         848x480       @ 90Hz     Z16
    Depth         848x480       @ 60Hz     Z16
    Depth         848x480       @ 30Hz     Z16
    Depth         848x480       @ 15Hz     Z16
    Depth         848x480       @ 6Hz      Z16
    Depth         848x100       @ 100Hz    Z16
    Depth         640x480       @ 90Hz     Z16
    Depth         640x480       @ 60Hz     Z16
    Depth         640x480       @ 30Hz     Z16
    Depth         640x480       @ 15Hz     Z16
    Depth         640x480       @ 6Hz      Z16
    Depth         640x360       @ 90Hz     Z16
    Depth         640x360       @ 60Hz     Z16
    Depth         640x360       @ 30Hz     Z16
    Depth         640x360       @ 15Hz     Z16
    Depth         640x360       @ 6Hz      Z16
    Depth         480x270       @ 90Hz     Z16
    Depth         480x270       @ 60Hz     Z16
    Depth         480x270       @ 30Hz     Z16
    Depth         480x270       @ 15Hz     Z16
    Depth         480x270       @ 6Hz      Z16
    Depth         424x240       @ 90Hz     Z16
    Depth         424x240       @ 60Hz     Z16
    Depth         424x240       @ 30Hz     Z16
    Depth         424x240       @ 15Hz     Z16
    Depth         424x240       @ 6Hz      Z16

Stream Profiles supported by RGB Camera
 Supported modes:
    stream       resolution      fps       format
    Color         1920x1080     @ 30Hz     RGB8
    Color         1920x1080     @ 30Hz     RAW16
    Color         1920x1080     @ 30Hz     Y16
    Color         1920x1080     @ 30Hz     BGRA8
    Color         1920x1080     @ 30Hz     RGBA8
    Color         1920x1080     @ 30Hz     BGR8
    Color         1920x1080     @ 30Hz     YUYV
    Color         1920x1080     @ 15Hz     RGB8
    Color         1920x1080     @ 15Hz     Y16
    Color         1920x1080     @ 15Hz     BGRA8
    Color         1920x1080     @ 15Hz     RGBA8
    Color         1920x1080     @ 15Hz     BGR8
    Color         1920x1080     @ 15Hz     YUYV
    Color         1920x1080     @ 6Hz      RGB8
    Color         1920x1080     @ 6Hz      Y16
    Color         1920x1080     @ 6Hz      BGRA8
    Color         1920x1080     @ 6Hz      RGBA8
    Color         1920x1080     @ 6Hz      BGR8
    Color         1920x1080     @ 6Hz      YUYV
    Color         1280x720      @ 30Hz     RGB8
    Color         1280x720      @ 30Hz     Y16
    Color         1280x720      @ 30Hz     BGRA8
    Color         1280x720      @ 30Hz     RGBA8
    Color         1280x720      @ 30Hz     BGR8
    Color         1280x720      @ 30Hz     YUYV
    Color         1280x720      @ 15Hz     RGB8
    Color         1280x720      @ 15Hz     Y16
    Color         1280x720      @ 15Hz     BGRA8
    Color         1280x720      @ 15Hz     RGBA8
    Color         1280x720      @ 15Hz     BGR8
    Color         1280x720      @ 15Hz     YUYV
    Color         1280x720      @ 6Hz      RGB8
    Color         1280x720      @ 6Hz      Y16
    Color         1280x720      @ 6Hz      BGRA8
    Color         1280x720      @ 6Hz      RGBA8
    Color         1280x720      @ 6Hz      BGR8
    Color         1280x720      @ 6Hz      YUYV
    Color         960x540       @ 60Hz     RGB8
    Color         960x540       @ 60Hz     Y16
    Color         960x540       @ 60Hz     BGRA8
    Color         960x540       @ 60Hz     RGBA8
    Color         960x540       @ 60Hz     BGR8
    Color         960x540       @ 60Hz     YUYV
    Color         960x540       @ 30Hz     RGB8
    Color         960x540       @ 30Hz     Y16
    Color         960x540       @ 30Hz     BGRA8
    Color         960x540       @ 30Hz     RGBA8
    Color         960x540       @ 30Hz     BGR8
    Color         960x540       @ 30Hz     YUYV
    Color         960x540       @ 15Hz     RGB8
    Color         960x540       @ 15Hz     Y16
    Color         960x540       @ 15Hz     BGRA8
    Color         960x540       @ 15Hz     RGBA8
    Color         960x540       @ 15Hz     BGR8
    Color         960x540       @ 15Hz     YUYV
    Color         960x540       @ 6Hz      RGB8
    Color         960x540       @ 6Hz      Y16
    Color         960x540       @ 6Hz      BGRA8
    Color         960x540       @ 6Hz      RGBA8
    Color         960x540       @ 6Hz      BGR8
    Color         960x540       @ 6Hz      YUYV
    Color         848x480       @ 60Hz     RGB8
    Color         848x480       @ 60Hz     Y16
    Color         848x480       @ 60Hz     BGRA8
    Color         848x480       @ 60Hz     RGBA8
    Color         848x480       @ 60Hz     BGR8
    Color         848x480       @ 60Hz     YUYV
    Color         848x480       @ 30Hz     RGB8
    Color         848x480       @ 30Hz     Y16
    Color         848x480       @ 30Hz     BGRA8
    Color         848x480       @ 30Hz     RGBA8
    Color         848x480       @ 30Hz     BGR8
    Color         848x480       @ 30Hz     YUYV
    Color         848x480       @ 15Hz     RGB8
    Color         848x480       @ 15Hz     Y16
    Color         848x480       @ 15Hz     BGRA8
    Color         848x480       @ 15Hz     RGBA8
    Color         848x480       @ 15Hz     BGR8
    Color         848x480       @ 15Hz     YUYV
    Color         848x480       @ 6Hz      RGB8
    Color         848x480       @ 6Hz      Y16
    Color         848x480       @ 6Hz      BGRA8
    Color         848x480       @ 6Hz      RGBA8
    Color         848x480       @ 6Hz      BGR8
    Color         848x480       @ 6Hz      YUYV
    Color         640x480       @ 60Hz     RGB8
    Color         640x480       @ 60Hz     Y16
    Color         640x480       @ 60Hz     BGRA8
    Color         640x480       @ 60Hz     RGBA8
    Color         640x480       @ 60Hz     BGR8
    Color         640x480       @ 60Hz     YUYV
    Color         640x480       @ 30Hz     RGB8
    Color         640x480       @ 30Hz     Y16
    Color         640x480       @ 30Hz     BGRA8
    Color         640x480       @ 30Hz     RGBA8
    Color         640x480       @ 30Hz     BGR8
    Color         640x480       @ 30Hz     YUYV
    Color         640x480       @ 15Hz     RGB8
    Color         640x480       @ 15Hz     Y16
    Color         640x480       @ 15Hz     BGRA8
    Color         640x480       @ 15Hz     RGBA8
    Color         640x480       @ 15Hz     BGR8
    Color         640x480       @ 15Hz     YUYV
    Color         640x480       @ 6Hz      RGB8
    Color         640x480       @ 6Hz      Y16
    Color         640x480       @ 6Hz      BGRA8
    Color         640x480       @ 6Hz      RGBA8
    Color         640x480       @ 6Hz      BGR8
    Color         640x480       @ 6Hz      YUYV
    Color         640x360       @ 60Hz     RGB8
    Color         640x360       @ 60Hz     Y16
    Color         640x360       @ 60Hz     BGRA8
    Color         640x360       @ 60Hz     RGBA8
    Color         640x360       @ 60Hz     BGR8
    Color         640x360       @ 60Hz     YUYV
    Color         640x360       @ 30Hz     RGB8
    Color         640x360       @ 30Hz     Y16
    Color         640x360       @ 30Hz     BGRA8
    Color         640x360       @ 30Hz     RGBA8
    Color         640x360       @ 30Hz     BGR8
    Color         640x360       @ 30Hz     YUYV
    Color         640x360       @ 15Hz     RGB8
    Color         640x360       @ 15Hz     Y16
    Color         640x360       @ 15Hz     BGRA8
    Color         640x360       @ 15Hz     RGBA8
    Color         640x360       @ 15Hz     BGR8
    Color         640x360       @ 15Hz     YUYV
    Color         640x360       @ 6Hz      RGB8
    Color         640x360       @ 6Hz      Y16
    Color         640x360       @ 6Hz      BGRA8
    Color         640x360       @ 6Hz      RGBA8
    Color         640x360       @ 6Hz      BGR8
    Color         640x360       @ 6Hz      YUYV
    Color         424x240       @ 60Hz     RGB8
    Color         424x240       @ 60Hz     Y16
    Color         424x240       @ 60Hz     BGRA8
    Color         424x240       @ 60Hz     RGBA8
    Color         424x240       @ 60Hz     BGR8
    Color         424x240       @ 60Hz     YUYV
    Color         424x240       @ 30Hz     RGB8
    Color         424x240       @ 30Hz     Y16
    Color         424x240       @ 30Hz     BGRA8
    Color         424x240       @ 30Hz     RGBA8
    Color         424x240       @ 30Hz     BGR8
    Color         424x240       @ 30Hz     YUYV
    Color         424x240       @ 15Hz     RGB8
    Color         424x240       @ 15Hz     Y16
    Color         424x240       @ 15Hz     BGRA8
    Color         424x240       @ 15Hz     RGBA8
    Color         424x240       @ 15Hz     BGR8
    Color         424x240       @ 15Hz     YUYV
    Color         424x240       @ 6Hz      RGB8
    Color         424x240       @ 6Hz      Y16
    Color         424x240       @ 6Hz      BGRA8
    Color         424x240       @ 6Hz      RGBA8
    Color         424x240       @ 6Hz      BGR8
    Color         424x240       @ 6Hz      YUYV
    Color         320x240       @ 60Hz     RGB8
    Color         320x240       @ 60Hz     Y16
    Color         320x240       @ 60Hz     BGRA8
    Color         320x240       @ 60Hz     RGBA8
    Color         320x240       @ 60Hz     BGR8
    Color         320x240       @ 60Hz     YUYV
    Color         320x240       @ 30Hz     RGB8
    Color         320x240       @ 30Hz     Y16
    Color         320x240       @ 30Hz     BGRA8
    Color         320x240       @ 30Hz     RGBA8
    Color         320x240       @ 30Hz     BGR8
    Color         320x240       @ 30Hz     YUYV
    Color         320x240       @ 6Hz      RGB8
    Color         320x240       @ 6Hz      Y16
    Color         320x240       @ 6Hz      BGRA8
    Color         320x240       @ 6Hz      RGBA8
    Color         320x240       @ 6Hz      BGR8
    Color         320x240       @ 6Hz      YUYV
    Color         320x180       @ 60Hz     RGB8
    Color         320x180       @ 60Hz     Y16
    Color         320x180       @ 60Hz     BGRA8
    Color         320x180       @ 60Hz     RGBA8
    Color         320x180       @ 60Hz     BGR8
    Color         320x180       @ 60Hz     YUYV
    Color         320x180       @ 30Hz     RGB8
    Color         320x180       @ 30Hz     Y16
    Color         320x180       @ 30Hz     BGRA8
    Color         320x180       @ 30Hz     RGBA8
    Color         320x180       @ 30Hz     BGR8
    Color         320x180       @ 30Hz     YUYV
    Color         320x180       @ 6Hz      RGB8
    Color         320x180       @ 6Hz      Y16
    Color         320x180       @ 6Hz      BGRA8
    Color         320x180       @ 6Hz      RGBA8
    Color         320x180       @ 6Hz      BGR8
    Color         320x180       @ 6Hz      YUYV

Stream Profiles supported by Motion Module
 Supported modes:
    stream       resolution      fps       format
    Gyro         N/A            @ 200Hz    MOTION_XYZ32F
    Gyro         N/A            @ 400Hz    MOTION_XYZ32F
    Accel        N/A            @ 63Hz     MOTION_XYZ32F
    Accel        N/A            @ 250Hz    MOTION_XYZ32F

Intrinsic Parameters:

 Intrinsic of "Depth" / 424x240 / {Z16}
  Width:        424
  Height:       240
  PPX:          210.147827148438
  PPY:          117.526298522949
  Fx:           210.793334960938
  Fy:           210.793334960938
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 480x270 / {Z16}
  Width:        480
  Height:       270
  PPX:          237.903198242188
  PPY:          132.149185180664
  Fx:           238.63395690918
  Fy:           238.63395690918
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 640x360 / {Z16}
  Width:        640
  Height:       360
  PPX:          317.204284667969
  PPY:          176.26611328125
  Fx:           318.178619384766
  Fy:           318.178619384766
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 640x480 / {Z16}
  Width:        640
  Height:       480
  PPX:          316.645141601563
  PPY:          235.519332885742
  Fx:           381.814331054688
  Fy:           381.814331054688
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 848x100 / {Z16}
  Width:        848
  Height:       100
  PPX:          420.295654296875
  PPY:          45.0525970458984
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 848x480 / {Z16}
  Width:        848
  Height:       480
  PPX:          420.295654296875
  PPY:          235.052597045898
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Depth" / 1280x720 / {Z16}
  Width:        1280
  Height:       720
  PPX:          634.408569335938
  PPY:          352.5322265625
  Fx:           636.357238769531
  Fy:           636.357238769531
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 320x180 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        320
  Height:       180
  PPX:          157.844299316406
  PPY:          89.1665802001953
  Fx:           229.652008056641
  Fy:           228.993255615234
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 320x240 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        320
  Height:       240
  PPX:          157.125732421875
  PPY:          118.888778686523
  Fx:           306.202667236328
  Fy:           305.324340820313
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 424x240 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        424
  Height:       240
  PPX:          209.125732421875
  PPY:          118.888778686523
  Fx:           306.202667236328
  Fy:           305.324340820313
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 640x360 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        640
  Height:       360
  PPX:          315.688598632813
  PPY:          178.333160400391
  Fx:           459.304016113281
  Fy:           457.986511230469
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 640x480 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        640
  Height:       480
  PPX:          314.25146484375
  PPY:          237.777557373047
  Fx:           612.405334472656
  Fy:           610.648681640625
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 848x480 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        848
  Height:       480
  PPX:          418.25146484375
  PPY:          237.777557373047
  Fx:           612.405334472656
  Fy:           610.648681640625
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 960x540 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        960
  Height:       540
  PPX:          473.532928466797
  PPY:          267.499755859375
  Fx:           688.955993652344
  Fy:           686.979736328125
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 1280x720 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16}
  Width:        1280
  Height:       720
  PPX:          631.377197265625
  PPY:          356.666320800781
  Fx:           918.608032226563
  Fy:           915.973022460938
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Color" / 1920x1080 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y16/RAW16}
  Width:        1920
  Height:       1080
  PPX:          947.065856933594
  PPY:          534.99951171875
  Fx:           1377.91198730469
  Fy:           1373.95947265625
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 424x240 / {Y8}
  Width:        424
  Height:       240
  PPX:          210.147827148438
  PPY:          117.526298522949
  Fx:           210.793334960938
  Fy:           210.793334960938
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 480x270 / {Y8}
  Width:        480
  Height:       270
  PPX:          237.903198242188
  PPY:          132.149185180664
  Fx:           238.63395690918
  Fy:           238.63395690918
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 640x360 / {Y8}
  Width:        640
  Height:       360
  PPX:          317.204284667969
  PPY:          176.26611328125
  Fx:           318.178619384766
  Fy:           318.178619384766
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 640x480 / {Y8}
  Width:        640
  Height:       480
  PPX:          316.645141601563
  PPY:          235.519332885742
  Fx:           381.814331054688
  Fy:           381.814331054688
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 848x100 / {Y8}
  Width:        848
  Height:       100
  PPX:          420.295654296875
  PPY:          45.0525970458984
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 848x480 / {Y8}
  Width:        848
  Height:       480
  PPX:          420.295654296875
  PPY:          235.052597045898
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 1280x720 / {Y8}
  Width:        1280
  Height:       720
  PPX:          634.408569335938
  PPY:          352.5322265625
  Fx:           636.357238769531
  Fy:           636.357238769531
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 1" / 1280x800 / {Y8}
  Width:        1280
  Height:       800
  PPX:          634.408569335938
  PPY:          392.5322265625
  Fx:           636.357238769531
  Fy:           636.357238769531
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 424x240 / {Y8}
  Width:        424
  Height:       240
  PPX:          210.147827148438
  PPY:          117.526298522949
  Fx:           210.793334960938
  Fy:           210.793334960938
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 480x270 / {Y8}
  Width:        480
  Height:       270
  PPX:          237.903198242188
  PPY:          132.149185180664
  Fx:           238.63395690918
  Fy:           238.63395690918
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 640x360 / {Y8}
  Width:        640
  Height:       360
  PPX:          317.204284667969
  PPY:          176.26611328125
  Fx:           318.178619384766
  Fy:           318.178619384766
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 640x480 / {Y8}
  Width:        640
  Height:       480
  PPX:          316.645141601563
  PPY:          235.519332885742
  Fx:           381.814331054688
  Fy:           381.814331054688
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 848x100 / {Y8}
  Width:        848
  Height:       100
  PPX:          420.295654296875
  PPY:          45.0525970458984
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 848x480 / {Y8}
  Width:        848
  Height:       480
  PPX:          420.295654296875
  PPY:          235.052597045898
  Fx:           421.586669921875
  Fy:           421.586669921875
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 1280x720 / {Y8}
  Width:        1280
  Height:       720
  PPX:          634.408569335938
  PPY:          352.5322265625
  Fx:           636.357238769531
  Fy:           636.357238769531
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

 Intrinsic of "Infrared 2" / 1280x800 / {Y8}
  Width:        1280
  Height:       800
  PPX:          634.408569335938
  PPY:          392.5322265625
  Fx:           636.357238769531
  Fy:           636.357238769531
  Distortion:   Brown Conrady
  Coeffs:       0       0       0       0       0

Motion Intrinsic Parameters:

Motion Intrinsic of "Gyro"        MOTION_XYZ32F
Bias Variances:         0.000000000000000  0.000000000000000  0.000000000000000
Noise Variances:        0.000000000000000  0.000000000000000  0.000000000000000
Sensitivity :
     1.000000       0.000000       0.000000      -0.000015
     0.000000       1.000000       0.000000       0.000053
     0.000000       0.000000       1.000000      -0.000011

Motion Intrinsic of "Accel"       MOTION_XYZ32F
Bias Variances:         0.000000000000000  0.000000000000000  0.000000000000000
Noise Variances:        0.000000000000000  0.000000000000000  0.000000000000000
Sensitivity :
     1.017338       0.000683      -0.012806       0.118170
    -0.034533       1.018766       0.009339      -0.236348
    -0.011902       0.006347       1.022751       0.206691

Extrinsic Parameters:
Extrinsic from "Depth"    To      "Depth" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Depth"    To      "Color" :
N/A
Extrinsic from "Depth"    To      "Infrared 1" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Depth"    To      "Infrared 2" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.0500018894672394  0  0

Extrinsic from "Depth"    To      "Gyro" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.00552000012248755  0.00510000018402934  0.011739999987185

Extrinsic from "Depth"    To      "Accel" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.00552000012248755  0.00510000018402934  0.011739999987185

Extrinsic from "Color"    To      "Depth" :
N/A
Extrinsic from "Color"    To      "Color" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Color"    To      "Infrared 1" :
N/A
Extrinsic from "Color"    To      "Infrared 2" :
N/A
Extrinsic from "Color"    To      "Gyro" :
N/A
Extrinsic from "Color"    To      "Accel" :
N/A
Extrinsic from "Infrared 1"       To      "Depth" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Infrared 1"       To      "Color" :
N/A
Extrinsic from "Infrared 1"       To      "Infrared 1" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Infrared 1"       To      "Infrared 2" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.0500018894672394  0  0

Extrinsic from "Infrared 1"       To      "Gyro" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.00552000012248755  0.00510000018402934  0.011739999987185

Extrinsic from "Infrared 1"       To      "Accel" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.00552000012248755  0.00510000018402934  0.011739999987185

Extrinsic from "Infrared 2"       To      "Depth" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.0500018894672394  0  0

Extrinsic from "Infrared 2"       To      "Color" :
N/A
Extrinsic from "Infrared 2"       To      "Infrared 1" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.0500018894672394  0  0

Extrinsic from "Infrared 2"       To      "Infrared 2" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Infrared 2"       To      "Gyro" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.0444818884134293  0.00510000018402934  0.011739999987185

Extrinsic from "Infrared 2"       To      "Accel" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.0444818884134293  0.00510000018402934  0.011739999987185

Extrinsic from "Gyro"     To      "Depth" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.00552000012248755  -0.00510000018402934  -0.011739999987185

Extrinsic from "Gyro"     To      "Color" :
N/A
Extrinsic from "Gyro"     To      "Infrared 1" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.00552000012248755  -0.00510000018402934  -0.011739999987185

Extrinsic from "Gyro"     To      "Infrared 2" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.0444818884134293  -0.00510000018402934  -0.011739999987185

Extrinsic from "Gyro"     To      "Gyro" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Gyro"     To      "Accel" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Accel"    To      "Depth" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.00552000012248755  -0.00510000018402934  -0.011739999987185

Extrinsic from "Accel"    To      "Color" :
N/A
Extrinsic from "Accel"    To      "Infrared 1" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0.00552000012248755  -0.00510000018402934  -0.011739999987185

Extrinsic from "Accel"    To      "Infrared 2" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: -0.0444818884134293  -0.00510000018402934  -0.011739999987185

Extrinsic from "Accel"    To      "Gyro" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Extrinsic from "Accel"    To      "Accel" :
 Rotation Matrix:
   1   0   0
   0   1   0
   0   0   1

 Translation Vector: 0  0  0

Seems like the extrinsics between color and all other sensors are broken, however I did not receive any warnings/errors when using align.

jonblob commented 5 years ago

I just tested my code and rs-align using a colleague's D435, which returned properly aligned images. For that camera, the extrinsics between color and all other sensors as shown in rs-enumerate-devices -c also seem to be fine.

ev-mp commented 5 years ago

@jonblob hello, unfortunately this is the same issue as in #3788, #3949

Extrinsic from "Depth" To "Color" : N/A

In v2.22 we release a workaround that apparently worked only for some users. We'll return to you on this.

jonblob commented 5 years ago

I am using v2.22, so apparently the workaround does not work in my case. Thank you for your fast response though! I hope this issue can be resolved soon.

ev-mp commented 5 years ago

@jonblob hello,

Please create and share the camera dump generated with the attached python script realsense_flash_debug.txt

Instructions:

  1. Install python 2.7/3.6
  2. Install realsense python wrapper pip install pyrealsense2
  3. Rename the attached script extension: realsense_flash_debug.txt into realsense_flash_debug.py
  4. Run the script python realsense_flash_debug.py The script will run for ~2min and generate device FW dump realsense_debug_dump.bin
  5. Attach the resulted file
jonblob commented 5 years ago

Hello, I ran the script as described and attached the FW dump. I renamed the .bin to .txt to be able to attach it. realsense_debug_dump.txt

jonblob commented 5 years ago

Can you please tell me how long it will take until an update is released? Without being able to align color and depth images, I can not use the camera.

zhuzunjie17 commented 5 years ago

Exactly the same issue

xiaoyanluo commented 5 years ago

Exactly the same issue 我也是很多问题，你呢？求交流啊 WX Ai1616166

RealSenseCustomerSupport commented 5 years ago

@jonblob The latest librealsense v2.23.0 and FW 5.11.6.250 should fix the issue. Please have a try. Thanks!

FW link: https://downloadmirror.intel.com/28870/eng/D400_Series_Production_FW_5_11_6_250.zip

jonblob commented 5 years ago

Thank you for the update! I installed the firmware version 5.11.6.250 and built the new librealsense v2.23.0. Unfortunately, the issue is still not fixed. Both using rs-align and using my scripts with pyrealsense, the output of align is a black image (both for depth to color and color to depth). When executing rs-enumerate-devices or checking the extrinsics via pyrealsense, the extrinsics of all other streams to the color stream are still zeros, exactly the same as in my previous upload.

Are there any other things to try? Otherwise I would ask to return the camera or exchange it for a D435, since I can not proceed with my research without being able to align depth and color images.

RealSenseCustomerSupport commented 5 years ago

@jonblob Really sorry for the trouble! I'll further check what you can check in this situation and will update you early next week.

jonblob commented 5 years ago

Thank you very much!

RealSenseCustomerSupport commented 5 years ago

@jonblob

Please install Calibration API and try to restore golden calibration table by "Intel.Realsense.CustomRW.exe -g" to see if the issue can fix or not. If still can't fix, please exchange the camera with the distributor. Sorry for the trouble!

Calibration API download link: https://downloadmirror.intel.com/28517/eng/CalibrationAPI-2.6.8.0-Setup.zip User guide: https://www.intel.com/content/dam/support/us/en/documents/emerging-technologies/intel-realsense-technology/RealSense_D400_Dyn_Calib_User_Guide.pdf

jonblob commented 5 years ago

Thank you for your help. I ran "Intel.RealsenseCustomRW.exe -g", resulting in

CustomRW for Intel RealSense D400, Version: 2.6.8.0

  Device PID: 0B3A
  Device name: Intel RealSense D435I
  Serial number: ************
  Firmware version: 05.11.06.250

Calibration on device successfully reset to default gold factory settings.

Unfortunately, rs-enumerate-devices still shows "N/A" for the extrinsics between color and any other stream. rs-align and the align function in pyrealsense2 also do not work. If possible, I would like to exchange the camera for a D435 with correct extrinsic calibration so I can continue properly. The issue can then be closed.

RealSenseCustomerSupport commented 5 years ago

@jonblob Sorry to hear that the issue not fixed. Yes, please exchange the camera.