No rendered depth map in tto process

[TwiceMao]

@cmh1027 For the test set, when we executed the tto code, we found that the Depth or inverse depth map of the test set was not rendered. Moreover, we hope to directly save the rendered RGB images and Depth images as logs locally instead of WANDB on the network, because this can shorten the time. Could you please give me any advice and convenience?

[TwiceMao]

I would also like to ask you why the apearance embedding during tto was not saved? If I want to save., how should I do.

[cmh1027]

We optimize the appearance embedding from scratch in appearance TTO, so there was no need to save the embedding in pose TTO. If you want to save the embedding in pose TTO, you may just add this code (you can found it from line 210) in the if self.hparams["pose_optimize"] block in nerf_system_optimize.py np.save(save_path, np.array(self.embedding_fine_a(img_idx[0]).cpu()))

[cmh1027]

For depth image, you can refer to nerf_system.py

        for log_img_name in self.hparams["val.log_image_list"]:
            try:
                if "depth" in log_img_name:
                    depth = results[log_img_name].view(H, W)  # (3, H, W)
                    img = viz_utils.visualize_depth(depth)
                elif "feat" in log_img_name and feats is not None:
                    feat_map = results[log_img_name].view(H, W, -1)
                    img = viz_utils.get_pca_img(feat_map, pca_m, pca_c).permute(2, 0, 1)
                elif "rgb" in log_img_name:
                    img = (
                        results[log_img_name].view(H, W, -1).permute(2, 0, 1).cpu()
                    )  # (3, H, W)
                self.logger.log_image(f"val_{img_idx}/viz/{log_img_name}", [img])
            except:
                pass

You can add script in the [if "depth" in log_img_name] block.

[TwiceMao]

@cmh1027 When doing your tto, what will happen if the ground truth of all test poses is not aligned with the up-nerf and is directly initialized by the identity matrix? Do you think this would be fine?

[cmh1027]

I believe it will vary based on the pose. If it is too far from the identity matrix, I think it will fail.