Closed vlasu19 closed 1 month ago
Hi,
the issue is that you assume the depth and the confidence images have a certain hardcoded resolution (the RGB image is landscape, whereas the depth and conf. images are incorrectly portrait in the screenshot you shared).
Since Record3D 1.10, landscape video recording is possible, so the code you use to read the depth and confidence images is incorrect (it does not enumerate all possible combinations of resolutions, and it even assumes incorrect FaceID resolution. Moreover, confidence files are only available for LiDAR video.).
Instead of hardcoding the resolutions, I highly recommend to parse the metadata
JSON file in your script; that file is stored in every r3d
file. The width and height of the depth and confidence images (both have identical resolution) is stored in the JSON file under the keys dw
and dh
. The key cameraType
contains either 0
if the video was recorded with the selfie FaceID camera or 1
for LiDAR videos.
I've changed the hardcoded resolution issue by parsing the metadata, but the issue still exists.
self.depth_width = metadata_dict['dw']
self.depth_height = metadata_dict['dh']
depth_img = depth_img.reshape((self.depth_width, self.depth_height))
To align the depth image with the rgb image, I used this code process. This works on an older version of the r3d file (08/2022)
def _reshape_all_depth_and_conf(self):
for index in tqdm.trange(len(self.poses), desc="Upscaling depth and conf"):
depth_image = self._depth_images[index]
# Upscale depth image.
pil_img = Image.fromarray(depth_image)
reshaped_img = pil_img.resize((self.rgb_width, self.rgb_height))
reshaped_img = np.asarray(reshaped_img)
self._reshaped_depth.append(reshaped_img)
# Upscale confidence as well
confidence = self._confidences[index]
conf_img = Image.fromarray(confidence)
reshaped_conf = conf_img.resize((self.rgb_width, self.rgb_height))
reshaped_conf = np.asarray(reshaped_conf)
self._reshaped_conf.append(reshaped_conf)
I solved the problem, and the reason that the depth image and confidence image appeared to be different from the rgb direction was because of a wrong reshape of width and height. It was strange, but the width and height reshape in the code should be correct. Anyway, the problem was solved after I switched them. Thanks for you help.
depth_img = depth_img.reshape((self.depth_height, self.depth_width))
Hi! When I read the depth image and confidence image output from the r3d file, this happens, what is the reason for this and what is the solution. Am I doing some wrong in my apps settings?
The code to read the images is below: