Gibson Dataset glb files

[tayyabkhalil-313]

I have downloaded the gibson tiny dataset from here as gibson_tiny.tar.gz which contained some obj files and others for each scene but no glb files. I then downloaded the 3DSceneGraph and then converted it via tools/gen_gibson_semantics.sh /path/to/3DSceneGraph_medium/automated_graph /path/to/GibsonDataset /path/to/output which produced some scns and ids. But I need the glb files for the scenes to run the demo notebook and I don't understand how or where to obtain those. Any directions on how to get those glb files?

[Kenn3o3]

you can download them in the same google form here

Download the gibson dataset (may want to start with smaller one):

Inside the form:

Gibson Dataset (trainval) for use with Habitat (11 GB): All training and validation scenes of the Gibson dataset for use with Habitat Sim:
https://dl.fbaipublicfiles.com/habitat/data/scene_datasets/gibson_habitat_trainval.zip

Gibson Dataset for Habitat challenge (1.5 GB): a smaller subset of 4+ quality scenes (as described in the Habitat ICCV19 paper https://arxiv.org/abs/1904.01201):
https://dl.fbaipublicfiles.com/habitat/data/scene_datasets/gibson_habitat.zip

===

Also, I currently have trouble getting the object encoder to work ( the simulator cannot get semantic sensor input ). It keeps showing: [21:27:07:360364]:[Warning]:[Sim] Simulator.cpp(599)::instanceStageForSceneAttributes : The active scene does not contain semantic annotations : activeSemanticSceneID_ = 0

Could you let me know if you have fixed it and get the object encoder to work please? Thank you very much!

[tayyabkhalil-313]

Thanks. Once I get the demo working, I'll look into the object encoder. Were you able to make the demo notebook work directly or it needed some changes?

[Kenn3o3]

For some reason I am not able to get the cv2 library to work probably in jupyter notebook, so I put all the python code in a .py file and it works just fine except that object encoder is not working. demo.py:

import os
project_dir = '/home/user1/Desktop/quadrupedal-2024/code/habitat_ai/project_ws'
os.chdir(project_dir)

import torch
import torch
import cv2, glob, imageio, joblib
from PIL import Image
from tqdm import tqdm 
import quaternion as q
import matplotlib.pyplot as plt
import numpy as np
import habitat, habitat_sim
habitat_path = habitat.__path__[0]
from sklearn.metrics.pairwise import cosine_similarity
import torchvision.transforms as transforms
import torch.nn as nn
from utils.habitat_settings import default_sim_settings, make_cfg
from utils.statics import CATEGORIES, COI_INDEX
from matplotlib import animation
from IPython.display import HTML
from scipy.ndimage import rotate
from habitat.utils.visualizations import utils, fog_of_war
import imageio.v3 as iio
from torchvision.models import resnet18 as resnet18_img
from model.Graph.resnet_obj import resnet18 as resnet18_obj
from magnum import Quaternion, Vector3, Rad
def check_gpu_status():
    # Check if CUDA is available
    if torch.cuda.is_available():
        print("CUDA is available! 🎉")
        # Get the number of CUDA devices
        num_devices = torch.cuda.device_count()
        print(f"Number of GPU devices available: {num_devices}")
        for i in range(num_devices):
            print(f"Device {i}: {torch.cuda.get_device_name(i)}")
            # Additional device properties can be added here
            print("  - Memory Allocated:", torch.cuda.memory_allocated(i))
            print("  - Memory Cached:   ", torch.cuda.memory_reserved(i))
            print("  - Capability:      ", torch.cuda.get_device_capability(i))
            print("  - Device Properties:", torch.cuda.get_device_properties(i))
    else:
        print("CUDA is not available. 😢")

def get_object_feat(rgb, bboxes, num_of_camera=12):
    image = eval_augmentation(Image.fromarray(rgb.astype(np.uint8)))[None].cuda().float()
    feat = object_encoder(image, torch.tensor(np.concatenate((np.zeros([len(bboxes), 1]), bboxes), -1)).cuda()[None].float())[0]
    feat = nn.functional.normalize(feat, dim=1)
    return feat

def get_image_feat(rgb, depth, num_of_camera=12):
    img_tensor = (torch.tensor(rgb[None]).cuda().float() / 255.).permute(0, 3, 1, 2)
    feat = img_encoder(img_tensor)
    vis_embedding = nn.functional.normalize(feat, dim=1)
    return vis_embedding

class ImgGraph(object):
    def __init__(self):
        self.feature_dim = 512
        self.M = 100

    def num_node(self):
        return len(self.node_position_list)

    def reset(self):
        self.node_position_list = []  # This position list is only for visualizations
        self.node_rotation_list = []  # This rotation list is only for visualizations
        self.graph_memory = np.zeros([self.M, self.feature_dim])
        self.A = np.zeros([self.M, self.M], dtype=np.bool_)
        self.graph_mask = np.zeros(self.M)
        self.graph_time = np.zeros(self.M)
        self.last_localized_node_idx = np.zeros([1], dtype=np.int32)
        self.last_localized_node_embedding = np.zeros([self.feature_dim], dtype=np.float32)

    def initialize_graph(self, new_embeddings, positions, rotations):
        self.add_node(node_idx=0, embedding=new_embeddings, time_step=0, position=positions, rotation=rotations)
        self.record_localized_state(node_idx=0, embedding=new_embeddings)

    def add_node(self, node_idx, embedding, time_step, position, rotation):
        self.node_position_list.append(position)
        self.node_rotation_list.append(rotation)
        self.graph_memory[node_idx] = embedding
        self.graph_mask[node_idx] = 1.0
        self.graph_time[node_idx] = time_step

    def record_localized_state(self, node_idx, embedding):
        self.last_localized_node_idx = node_idx
        self.last_localized_node_embedding = embedding

    def add_edge(self, node_idx_a, node_idx_b):
        self.A[node_idx_a, node_idx_b] = True
        self.A[node_idx_b, node_idx_a] = True

    def update_node(self, node_idx, time_info, embedding=None):
        if embedding is not None:
            self.graph_memory[node_idx] = embedding
        self.graph_time[node_idx] = time_info

    def get_positions(self, a=None):
        if a is None:
            return self.node_position_list
        else:
            return self.node_position_list[a]

    def get_neighbor(self, node_idx, return_mask=False):
        if return_mask:
            return self.A[node_idx]
        else:
            return np.where(self.A[node_idx])[0]

class ObjGraph(object):
    def __init__(self,):
        self.feature_dim = 32
        self.M = 1000
        self.MV = 100

    def num_node(self):
        return len(self.node_position_list)

    def reset(self):
        self.node_position_list = []  # This position list is only for visualizations
        self.graph_memory = np.zeros([self.M, self.feature_dim])
        self.graph_category = np.zeros([self.M])
        self.graph_score = np.zeros([self.M])
        self.A_OV = np.zeros([self.M, self.MV], dtype=np.bool_)
        self.graph_mask = np.zeros(self.M)
        self.graph_time = np.zeros([self.M], dtype=np.int32)
        self.last_localized_node_idx = 0

    def initialize_graph(self, new_embeddings, object_scores, object_categories, masks, positions):
        if sum(masks == 1) == 0:
            masks[0] = 1
        self.add_node(node_idx=0, embedding=new_embeddings, object_score=object_scores, object_category=object_categories, time_step=0, mask=masks, position=positions, vis_node_idx=0)

    def add_node(self, node_idx, embedding, object_score, object_category, mask, time_step, position, vis_node_idx):
        node_idx_ = node_idx
        i = 0
        while True:
            if mask[i] == 1:
                self.node_position_list.append(position[i])
                self.graph_memory[node_idx_] = embedding[i]
                self.graph_score[node_idx_] = object_score[i]
                self.graph_category[node_idx_] = object_category[i]
                self.graph_mask[node_idx_] = 1.0
                self.graph_time[node_idx_] = time_step
                self.add_vo_edge([node_idx_], vis_node_idx)
                node_idx_ += 1
            i += 1
            if i == len(position):
                break

    def add_vo_edge(self, node_idx_obj, curr_vis_node_idx):
        for node_idx_obj_i in node_idx_obj:
            self.A_OV[node_idx_obj_i, curr_vis_node_idx] = True

    def update_node(self, node_idx, time_info, node_score, node_category, curr_vis_node_idx, embedding=None):
        if embedding is not None:
            self.graph_memory[node_idx] = embedding
        self.graph_score[node_idx] = node_score
        self.graph_category[node_idx] = node_category
        self.graph_time[node_idx] = time_info
        self.A_OV[node_idx, :] = False
        self.A_OV[node_idx, curr_vis_node_idx] = True

    def get_positions(self, a=None):
        if a is None:
            return self.node_position_list
        else:
            return self.node_position_list[a]

def is_close(embed_a, embed_b, return_prob=False, th=0.75):
    logits = np.matmul(embed_a, embed_b.transpose(1, 0))
    close = (logits > th)
    if return_prob:
        return close, logits
    else:
        return close

def update_image_graph(imggraph, objgraph, new_embedding, curr_obj_embeding, 
                       object_score, object_category, position, rotation, time, done, img_node_th=0.8, obj_node_th=0.8):
    # The position is only used for visualizations
    if done:
        imggraph.reset()
        imggraph.initialize_graph(new_embedding, position, rotation)

    obj_close = True
    obj_graph_mask = objgraph.graph_score[objgraph.A_OV[:, imggraph.last_localized_node_idx]] > 0.5
    if len(obj_graph_mask) > 0:
        curr_obj_mask = object_score > 0.5
        if np.sum(curr_obj_mask) / len(curr_obj_mask) >= 0.5:
            close_obj, prob_obj = is_close(objgraph.graph_memory[objgraph.A_OV[:, imggraph.last_localized_node_idx]], curr_obj_embeding, return_prob=True, th=obj_node_th)
            close_obj = close_obj[obj_graph_mask, :][:, curr_obj_mask]
            category_mask = objgraph.graph_category[objgraph.A_OV[:, imggraph.last_localized_node_idx]][obj_graph_mask][:, None] == object_category[curr_obj_mask]
            close_obj[~category_mask] = False
            if len(close_obj) >= 3:
                clos_obj_p = close_obj.any(1).sum() / (close_obj.shape[0])
                if clos_obj_p < 0.1:  # Fail to localize (find the same object) with the last localized frame
                    obj_close = False

    close, prob = is_close(imggraph.last_localized_node_embedding[None], new_embedding[None], return_prob=True, th=img_node_th)

    found = (np.array(done) + close.squeeze()) & np.array(obj_close).squeeze()
    found_prev = False
    found = found
    found_in_memory = False
    to_add = False
    if found:
        imggraph.update_node(imggraph.last_localized_node_idx, time)
        found_prev = True
    else:
        check_list = 1 - imggraph.graph_mask[:imggraph.num_node()]
        check_list[imggraph.last_localized_node_idx] = 1.0
        while not found:
            not_checked_yet = np.where((1 - check_list))[0]
            neighbor_embedding = imggraph.graph_memory[not_checked_yet]
            num_to_check = len(not_checked_yet)
            if num_to_check == 0:
                to_add = True
                break
            else:
                close, prob = is_close(new_embedding[None], neighbor_embedding, return_prob=True, th=img_node_th)
                close = close[0];
                prob = prob[0]
                close_idx = np.where(close)[0]
                if len(close_idx) >= 1:
                    found_node = not_checked_yet[prob.argmax()]
                else:
                    found_node = None
                if found_node is not None:
                    found = True
                    if abs(time - imggraph.graph_time[found_node]) > 20:
                        found_in_memory = True
                    imggraph.update_node(found_node, time, new_embedding)
                    imggraph.add_edge(found_node, imggraph.last_localized_node_idx)
                    imggraph.record_localized_state(found_node, new_embedding)
                check_list[found_node] = 1.0

    if to_add:
        new_node_idx = imggraph.num_node()
        imggraph.add_node(new_node_idx, new_embedding, time, position, rotation)
        imggraph.add_edge(new_node_idx, imggraph.last_localized_node_idx)
        imggraph.record_localized_state(new_node_idx, new_embedding)
    last_localized_node_idx = imggraph.last_localized_node_idx
    return imggraph

def update_object_graph(imggraph, objgraph, object_embedding, object_score, object_category, object_mask, object_position, time, done, obj_node_th=0.8):
    # The position is only used for visualizations. Remove if object features are similar
    # object masking
    object_score = object_score[object_mask == 1]
    object_category = object_category[object_mask == 1]
    object_position = object_position[object_mask == 1]
    object_embedding = object_embedding[object_mask == 1]
    object_mask = object_mask[object_mask == 1]
    if done:
        objgraph.reset()
        objgraph.initialize_graph(object_embedding, object_score, object_category, object_mask, object_position)

    to_add = [True] * int(sum(object_mask))
    not_found = not done  # Dense connection mode
    if not_found:
        hop1_vis_node = imggraph.A[imggraph.last_localized_node_idx]
        hop1_obj_node_mask = np.sum(objgraph.A_OV.transpose(1, 0)[hop1_vis_node], 0) > 0
        curr_obj_node_mask = objgraph.A_OV[:, imggraph.last_localized_node_idx]
        neighbor_obj_node_mask = (hop1_obj_node_mask + curr_obj_node_mask) > 0
        neighbor_node_embedding = objgraph.graph_memory[neighbor_obj_node_mask]
        neighbor_obj_memory_idx = np.where(neighbor_obj_node_mask)[0]
        neighbor_obj_memory_score = objgraph.graph_score[neighbor_obj_memory_idx]
        neighbor_obj_memory_cat = objgraph.graph_category[neighbor_obj_memory_idx]

        close, prob = is_close(neighbor_node_embedding, object_embedding, return_prob=True, th=obj_node_th)
        for c_i in range(prob.shape[1]):
            close_mem_indices = np.where(close[:, c_i] == 1)[0]
            for m_i in close_mem_indices:
                is_same = False
                to_update = False
                if (object_category[c_i] == neighbor_obj_memory_cat[m_i]) and object_category[c_i] != -1:
                    is_same = True
                    if object_score[c_i] > neighbor_obj_memory_score[m_i]:
                        to_update = True

                if is_same:
                    to_add[c_i] = False

                if to_update:
                    objgraph.update_node(m_i, time, object_score[c_i], object_category[c_i], int(imggraph.last_localized_node_idx), object_embedding[c_i])
                    break

        # Add new objects to graph
        if sum(to_add) > 0:
            start_node_idx = objgraph.num_node()
            new_idx = np.where(np.stack(to_add))[0]
            objgraph.add_node(start_node_idx, object_embedding[new_idx], object_score[new_idx],
                              object_category[new_idx], object_mask[new_idx], time,
                              object_position[new_idx], int(imggraph.last_localized_node_idx))
    return objgraph

def draw_graph_on_map(topdownmap, imggraph, objgraph, draw_im_graph=True, draw_obj_graph=True, node_size_px=60):
    node_list = imggraph.node_position_list
    affinity = imggraph.A
    last_localized_imnode = imggraph.last_localized_node_idx
    print(last_localized_imnode)
    obj_node_list = objgraph.node_position_list
    ov_affinity = objgraph.A_OV
    if draw_obj_graph:
        h, w, _ = topdownmap.shape
        draw_obj_point_list = []
        for idx, node_position in enumerate(obj_node_list):
            connected_imnode = np.where(ov_affinity[idx])[0]
            draw_obj_point_list.append(node_position)

            if draw_im_graph:
                neighbors = np.where(ov_affinity[idx])[0]
                for neighbor_idx in neighbors:
                    neighbor_position = node_list[neighbor_idx]
                    node_map_pose = to_grid(node_position[0], node_position[2])
                    neighbor_map_pose = to_grid(neighbor_position[0], neighbor_position[2])
                    line = cv2.line(
                        topdownmap,
                        node_map_pose,
                        neighbor_map_pose,
                        [253, 173, 211],
                        thickness=10
                    )
                    alpha = 0.8
                    topdownmap = cv2.addWeighted(line, alpha, topdownmap, 1 - alpha, 0)

    if draw_im_graph:
        draw_im_point_list = []
        for idx, node_position in enumerate(node_list):
            if idx == last_localized_imnode:
                draw_im_point_list.append([node_position, [51, 153, 102]])
            else:
                draw_im_point_list.append([node_position, [15, 119, 143]])
            neighbors = np.where(affinity[idx])[0]
            for neighbor_idx in neighbors:
                neighbor_position = node_list[neighbor_idx]
                node_map_pose = to_grid(node_position[0], node_position[2])
                neighbor_map_pose = to_grid(neighbor_position[0], neighbor_position[2])
                cv2.line(
                    topdownmap,
                    node_map_pose,
                    neighbor_map_pose,
                    [177, 232, 246],
                    thickness=30,
                )

    if draw_im_graph:
        for node_position, node_color in draw_im_point_list:
            graph_node_center = to_grid(node_position[0], node_position[2])
            cv2.circle(topdownmap, graph_node_center, node_size_px // 3, node_color, -1)

    if draw_obj_graph:
        cnt = 0
        for node_position in draw_obj_point_list:
            graph_node_center = to_grid(node_position[0], node_position[2])
            triangle_cnt = np.array(graph_node_center + np.array([pt1, pt2, pt3]).astype(np.int32))
            cv2.drawContours(topdownmap, [triangle_cnt], 0, [248, 106, 176], -1)
            cnt += 1

    return topdownmap

def get_objects(semantic, mapping, depth, img_width, img_height):
    semantic = semantic.astype(np.int32)
    max_key = np.max(np.array(list(mapping.keys())))
    replace_values = []
    for i in np.arange(max_key + 1):
        try:
            replace_values.append(mapping[i])
        except:
            replace_values.append(-1)
    semantic_obs_class = np.take(replace_values, semantic)
    COI_MASK = [(semantic_obs_class == ci).astype(np.int32) for ci in COI_INDEX[dataset.split("_")[0]]]  # class mask
    unique_instances = np.unique(semantic * np.sum(np.stack(COI_MASK), 0))[1:]
    bbox_ids = unique_instances
    instance_segment = [(semantic == i).astype(np.int32) for i in unique_instances]
    box_categories = [np.unique(semantic_obs_class[semantic == i])[0] for i in unique_instances]
    if len(instance_segment) > 0:
        object_size = np.stack([np.sum(instance_segman) for instance_segman in instance_segment])
        mask = (object_size > 100)
        instance_segment = [instance_segman for i, instance_segman in enumerate(instance_segment) if mask[i] == 1]
        box_categories = np.stack(box_categories)[mask == 1]
        bbox_ids = np.array(bbox_ids)[mask == 1]

    x1s = [np.min(np.where(instance_segment[i])[1]) for i in range(len(instance_segment))]
    y1s = [np.min(np.where(instance_segment[i])[0]) for i in range(len(instance_segment))]
    x2s = [np.max(np.where(instance_segment[i])[1]) for i in range(len(instance_segment))]
    y2s = [np.max(np.where(instance_segment[i])[0]) for i in range(len(instance_segment))]
    bboxes = np.stack((x1s, y1s, x2s, y2s), 1)
    if len(bboxes) > 0:
        edge_box_idx = np.where(bboxes[:, 2] - bboxes[:, 0] > img_width * 0.8)[0]
        not_edge_box_idx = np.where(bboxes[:, 2] - bboxes[:, 0] <= img_width * 0.8)[0]
        if len(edge_box_idx) > 0:
            x1s1 = [np.min(np.where(instance_segment[i][:, :int(instance_segment[i].shape[1] / 2)])[1]) for i in edge_box_idx]
            y1s1 = [np.min(np.where(instance_segment[i][:, :int(instance_segment[i].shape[1] / 2)])[0]) for i in edge_box_idx]
            x2s1 = [np.max(np.where(instance_segment[i][:, :int(instance_segment[i].shape[1] / 2)])[1]) for i in edge_box_idx]
            y2s1 = [np.max(np.where(instance_segment[i][:, :int(instance_segment[i].shape[1] / 2)])[0]) for i in edge_box_idx]
            bboxes_1 = np.stack((x1s1, y1s1, x2s1, y2s1), 1)
            bboxes_1_categories = box_categories[edge_box_idx]
            bboxes_1_ids = bbox_ids[edge_box_idx]
            x1s2 = [int(instance_segment[i].shape[1] / 2) + np.min(np.where(instance_segment[i][:, int(instance_segment[i].shape[1] / 2):])[1]) for i in edge_box_idx]
            y1s2 = [np.min(np.where(instance_segment[i][:, int(instance_segment[i].shape[1] / 2):])[0]) for i in edge_box_idx]
            x2s2 = [int(instance_segment[i].shape[1] / 2) + np.max(np.where(instance_segment[i][:, int(instance_segment[i].shape[1] / 2):])[1]) for i in edge_box_idx]
            y2s2 = [np.max(np.where(instance_segment[i][:, int(instance_segment[i].shape[1] / 2):])[0]) for i in edge_box_idx]
            bboxes_2 = np.stack((x1s2, y1s2, x2s2, y2s2), 1)
            bboxes_2_categories = box_categories[edge_box_idx]
            bboxes_2_ids = bbox_ids[edge_box_idx]
            bboxes_ = bboxes[not_edge_box_idx]
            box_categories_ = box_categories[not_edge_box_idx]
            bbox_ids_ = bbox_ids[not_edge_box_idx]
            bboxes = np.concatenate((bboxes_, bboxes_1, bboxes_2), 0)
            box_categories = np.concatenate((box_categories_, bboxes_1_categories, bboxes_2_categories), 0)
            bbox_ids = np.concatenate((bbox_ids_, bboxes_1_ids, bboxes_2_ids), 0)

    if len(depth) > 0 and len(bboxes) > 0:
        box_pix_xs, box_pix_ys = ((bboxes[:, 0])).astype(np.int32), ((bboxes[:, 1])).astype(np.int32)
        box_depth = np.stack([depth[box_pix_y, box_pix_x] for box_pix_x, box_pix_y in zip(box_pix_xs, box_pix_ys)])
        box_mask = box_depth < 5
        if np.sum(box_mask) == 0:
            return [], [], [], [], []
        bboxes = bboxes[box_mask]
        box_categories = box_categories[box_mask]
        bbox_ids = bbox_ids[box_mask]
        box_depth = box_depth[box_mask]
        box_world = np.stack([object_loc[bbox_id] for bbox_id in bbox_ids])

        bboxes = bboxes.astype(np.float32)
        bboxes[:, 0] = bboxes[:, 0].astype(np.float32) / float(img_width)
        bboxes[:, 1] = bboxes[:, 1].astype(np.float32) / float(img_height)
        bboxes[:, 2] = bboxes[:, 2].astype(np.float32) / float(img_width)
        bboxes[:, 3] = bboxes[:, 3].astype(np.float32) / float(img_height)
        return bboxes, box_categories, bbox_ids, box_depth, box_world
    return [], [], [], [], []

def to_grid(realworld_x: float,realworld_y: float,):
    A = [realworld_x-(worldWidth+2*world_min_width)/2, realworld_y-(worldHeight+2*world_min_height)/2, 1, 1]
    grid_x, grid_y = np.array([imgWidth/2, imgHeight/2]) * np.matmul(P, A)[:2] + np.array([imgWidth/2, imgHeight/2])
    return int(grid_x), int(grid_y)

def from_grid(grid_x: int,grid_y: int):
    realworld_x, realworld_y = np.matmul(np.linalg.inv(P), [(2 * grid_x - imgWidth)/imgWidth, (2 * grid_y - imgHeight)/imgHeight, 1, 1])[:2] \
                               + np.array([(worldWidth+2*world_min_width)/2, (worldHeight+2*world_min_height)/2])
    return realworld_x, realworld_y

from habitat.utils.geometry_utils import (
    quaternion_from_coeff,
    quaternion_rotate_vector,
)
from habitat.tasks.utils import cartesian_to_polar

def get_map_angle(ref_rotation):
    heading_vector = quaternion_rotate_vector(
        ref_rotation.inverse(), np.array([0, 0, -1])
    )

    phi = cartesian_to_polar(-heading_vector[2], heading_vector[0])[1]
    return np.array(phi)

def draw_bbox(rgb: np.ndarray, bboxes: np.ndarray, color = (178,193,118), is_detection=False) -> np.ndarray:
    for i, bbox in enumerate(bboxes):
        imgHeight, imgWidth, _ = rgb.shape
        cv2.rectangle(rgb, (int(bbox[0]*imgWidth), int(bbox[1]*imgHeight)), (int(bbox[2]*imgWidth), int(bbox[3]*imgHeight)), (255, 255, 0), 5)
    return rgb

def step(in_action):
    action_names = list(cfg.agents[settings["default_agent"]].action_space.keys())
    action = action_names[in_action]
    sim.step(action)

def draw_agent(topdownmap, agent_position, agent_rotation, jackal_radius_px=50):
    rotated_jackal = rotate(
        JACKAL_SPRITE, agent_rotation * 180 / np.pi + 90
    )
    initial_jackal_size = JACKAL_SPRITE.shape[0]
    new_size = rotated_jackal.shape[0]
    jackal_size_px = max(1, int(jackal_radius_px * 2 * new_size / initial_agent_size))
    resized_jackal = cv2.resize(
        rotated_jackal,
        (jackal_size_px, jackal_size_px),
        interpolation=cv2.INTER_LINEAR
    )
    topdownmap = utils.paste_overlapping_image(topdownmap, resized_jackal, agent_position)
    return topdownmap

def concat_vertical(img1, img2):
    img1_shape = img1.shape
    img2_shape = img2.shape
    if img1_shape[1] > img2_shape[1]:
        img2 = cv2.resize(img2, (img1_shape[1], int(img2_shape[0] / img2_shape[1] * img1_shape[1])))
    elif img1_shape[1] < img2_shape[1]:
        img1 = cv2.resize(img1, (img2_shape[1], int(img1_shape[0] / img1_shape[1] * img2_shape[1])))
    return np.concatenate((img1, img2), axis=0)

def init():
    im.set_data(topdownmaps[0])

def animate(i):
    im.set_data(topdownmaps[i])
    return im

if __name__ == "__main__":
    check_gpu_status()
    """
    Set the display port!!
    """
    # os.environ['DISPLAY'] = 'localhost:10.0'
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
    eval_augmentation = transforms.Compose([
        transforms.ToTensor(),
        normalize
    ])
    img_encoder = resnet18_img(num_classes=512)
    dim_mlp = img_encoder.fc.weight.shape[1]
    img_encoder.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), img_encoder.fc)
    ckpt_pth = os.path.join(project_dir, 'data/graph', 'Img_encoder.pth.tar')
    ckpt = torch.load(ckpt_pth, map_location='cpu')
    state_dict = {k[len('module.encoder_q.'):]: v for k, v in ckpt['state_dict'].items() if 'module.encoder_q.' in k}
    img_encoder.load_state_dict(state_dict)
    img_encoder.eval().cuda()
    feature_dim = 32
    object_encoder = resnet18_obj(num_classes=feature_dim)
    dim_mlp = object_encoder.fc.weight.shape[1]
    object_encoder.fc = nn.Sequential(nn.Linear(dim_mlp, dim_mlp), nn.ReLU(), object_encoder.fc)
    ckpt_pth = os.path.join(project_dir, 'data/graph', 'Obj_encoder.pth.tar')
    ckpt = torch.load(ckpt_pth, map_location='cpu')
    state_dict = {k[len('module.encoder_k.'):]: v for k, v in ckpt['state_dict'].items() if 'module.encoder_k.' in k}
    object_encoder.load_state_dict(state_dict)
    object_encoder.eval().cuda()
    settings = default_sim_settings.copy()
    settings["max_frames"] = 100
    settings["width"] = 256
    settings["height"] = 256
    settings["scene"] = "/home/user1/Desktop/quadrupedal-2024/code/habitat_ai/habitat-lab/habitat-lab/data/scene_datasets/gibson/Albertville.glb"
    settings["save_png"] = False  # args.save_png
    settings["sensor_height"] = 0.88
    settings["color_sensor"] = True
    settings["semantic_sensor"] = True
    settings["depth_sensor"] = True
    settings["print_semantic_scene"] = False
    settings["print_semantic_mask_stats"] = False
    settings["compute_shortest_path"] = False
    settings["compute_action_shortest_path"] = False
    settings["panoramic_sensor"] = True
    settings["seed"] = 2343
    settings["silent"] = False
    settings["enable_physics"] = False
    settings["draw_lidar"] = False
    settings["agent_radius"] = 0.1
    settings["agent_height"] = 1.2
    settings["multiview"] = False
    settings["hfov"] = 90
    settings["FORWARD_STEP_SIZE"] = 0.25
    settings["TURN_ANGLE"] = 10
    settings["tdv_height"] = 512
    settings["tdv_width"] = 512
    settings["allow_sliding"] = True
    num_of_camera = 12
    pano_img_height = settings["height"]//2
    img_height = pano_img_height
    img_width = float(img_height * 4)
    cam_width = float(pano_img_height * 4 // num_of_camera)
    settings["cam_width"] = int(cam_width)
    settings["pano_height"] = int(pano_img_height)

    dataset = 'gibson'
    scan_name = "Andover"
    if dataset == "mp3d":
        settings["scene"] = os.path.join(habitat_path, '../data/scene_datasets/{}/{}/{}.glb'.format(dataset, scan_name, scan_name))
    elif "gibson" in dataset:
        settings["scene"] = os.path.join(habitat_path, '../data/scene_datasets/{}/{}.glb'.format(dataset, scan_name))
    elif dataset == "hm3d":
        path = glob.glob(os.path.join(habitat_path, '../data/scene_datasets/{}/*/{}/{}.glb'.format(dataset, "*" + scan_name, scan_name)))[0]
        settings["scene"] = path
    cfg = make_cfg(settings)

    try:
        sim.close()
    except:
        pass

    sim = habitat_sim.Simulator(cfg)
    max_object = 10
    bounds = sim.pathfinder.get_bounds()
    scene_objects = sim.semantic_scene.objects
    if dataset == "mp3d":
        mapping = {int(obj.id.split("_")[-1]): obj.category.index() for obj in scene_objects if obj != None}
    elif dataset == "hm3d":
        mapping = {int(obj.id.split("_")[-1]): obj.category.index() for obj in scene_objects if obj != None}
    else:
        mapping = {int(obj.id.split("_")[-1]): int(np.where([obj.category.name() == cat for cat in CATEGORIES['gibson']])[0][0]) for obj in scene_objects if obj != None}

    set_floor = 0
    floor = -1
    num_try = 0
    render_configs = joblib.load(os.path.join(project_dir, f"data/floorplans/{dataset}_floorplans/render_config.pkl"))
    while floor != set_floor:
        init_position = sim.pathfinder.get_random_navigable_point()
        floor =  int(np.argmin([abs(float(render_configs[scan_name][i]['z_low']) - init_position[1]) for i in render_configs[scan_name].keys()]))
        num_try += 1
        if num_try > 100:
            print("Cannot find a valid position")
            break
    init_rotation = sim.get_agent(0).get_state().rotation.components
    scene_file = settings["scene"]

    sim.reset()
    agent = sim.initialize_agent(0)
    start_state = agent.get_state()
    if (start_state.position != init_position).any():
        start_state.position = init_position
        start_state.rotation = q.from_float_array(init_rotation)  # self.init_rotation #
        start_state.sensor_states = dict()  ## Initialize sensori
    agent.set_state(start_state)
    prev_position = agent.get_state().position
    prev_rotation = q.as_euler_angles(agent.state.rotation)[1]

    P = render_configs[scan_name][floor]['Projection']
    lower_bound, upper_bound = sim.pathfinder.get_bounds()
    imgWidth = round(float(render_configs[scan_name][floor]['width']))
    imgHeight = round(float(render_configs[scan_name][floor]['height']))
    world_min_width = float(render_configs[scan_name][floor]['x_low'])
    world_max_width = float(render_configs[scan_name][floor]['x_high'])
    world_min_height = float(render_configs[scan_name][floor]['y_low'])
    world_max_height = float(render_configs[scan_name][floor]['y_high'])
    worldWidth = abs(world_min_width) + abs(world_max_width)
    worldHeight = abs(world_min_height) + abs(world_max_height)

    print('====================================')
    print('scan_name: ', scan_name)
    print('floor: ', floor)
    print('lower_bound: ', lower_bound)
    print('upper_bound: ', upper_bound)
    print('imgWidth: ', imgWidth)
    print('imgHeight: ', imgHeight)

    ortho_map = cv2.imread(os.path.join(project_dir, f"data/floorplans/{dataset}_floorplans/rgb/{scan_name}_level_{floor}.png"))[...,::-1][...,:3]
    ortho_mask = cv2.imread(os.path.join(project_dir, f"data/floorplans/{dataset}_floorplans/mask/{scan_name}_level_{floor}.png"), 0)
    ortho_map[ortho_mask==0] = 255
    scene_objects = sim.semantic_scene.objects
    object_loc = {int(obj.id.split("_")[-1]): obj.aabb.center for obj in scene_objects if obj != None}
    size= 0.4
    pt1 = np.array([150-150, 100-150])*size
    pt2 = np.array([100-150, 200-150])*size
    pt3 = np.array([200-150, 200-150])*size
    pt1 = pt1.astype(np.int32)
    pt2 = pt2.astype(np.int32)
    pt2 = pt2.astype(np.int32)
    JACKAL_SPRITE = iio.imread(os.path.join(project_dir, "data/assets/maps_topdown_agent_sprite/jackal.png"))
    JACKAL_SPRITE = np.ascontiguousarray(np.flipud(JACKAL_SPRITE))
    initial_agent_size = JACKAL_SPRITE.shape[0]
    action_names = list(cfg.agents[settings["default_agent"]].action_space.keys())
    print(action_names)
    start_position = Vector3([from_grid(356, 535)[0], init_position[1], from_grid(356, 535)[1]])
    goal_position = Vector3([from_grid(187, 124)[0], init_position[1], from_grid(187, 124)[1]])
    agent = sim.initialize_agent(0)
    start_state = agent.get_state()
    start_state.position = start_position
    agent.set_state(start_state)

    imggraph = ImgGraph()
    objgraph = ObjGraph()
    positions = []
    rotations = []
    topdownmap = ortho_map.copy()
    obs = sim.get_sensor_observations()
    rgb = np.concatenate([obs['panoramic_rgb_part_sensor_%d' % (i)][:, :, :3] for i in range(num_of_camera)], 1)
    semantic_array = np.concatenate([obs['panoramic_semantic_part_sensor_%d' % (i)] for i in range(num_of_camera)], 1)
    depth_array = np.concatenate([obs['panoramic_depth_part_sensor_%d' % (i)] for i in range(num_of_camera)], 1)
    bboxes, bbox_categories, bbox_ids, bbox_depth, bbox_pose = get_objects(semantic_array, mapping, depth_array, rgb.shape[1], rgb.shape[0])

    position = sim.get_agent(0).get_state().position
    rotation = sim.get_agent(0).get_state().rotation.components
    cur_map_coord = to_grid(position[0], position[2])[::-1]
    cur_map_rot = get_map_angle(sim.get_agent(0).get_state().rotation)
    image_feat = get_image_feat(rgb, depth_array).cpu().detach().numpy()
    object_feat = []
    if len(bboxes) > 0:
        object_feat = get_object_feat(rgb, bboxes).cpu().detach().numpy()
    position = sim.get_agent(0).get_state().position
    rotation = sim.get_agent(0).get_state().rotation.components
    imggraph.reset()
    imggraph.initialize_graph(image_feat[0], position, rotation)
    objgraph.reset()
    if len(bboxes) > 0:
        objgraph.initialize_graph(object_feat, np.ones(len(bboxes)), bbox_categories,np.ones(len(bboxes)), bbox_pose)
    topdownmap = draw_agent(topdownmap, cur_map_coord, cur_map_rot)

    cv2.imshow(f'vis', rgb[:, :, [2, 1, 0]])
    cv2.imshow(f'graph', topdownmap[::2, ::2, [2, 1, 0]])

    topdownmaps = []
    images = []
    t = 0
    while True:
        topdownmap = ortho_map.copy()
        key = cv2.waitKey(0)
        if key == ord('w'):
            step(1)
        elif key == ord('a'):
            step(2)
        elif key == ord('d'):
            step(3)
        elif key == ord('q'):
            cv2.destroyAllWindows()
            plt.imshow(topdownmap[::2, ::2, [2, 1, 0]])
            break
        position = sim.get_agent(0).get_state().position
        rotation = sim.get_agent(0).get_state().rotation.components
        cur_map_coord = to_grid(position[0], position[2])[::-1]
        cur_map_rot = get_map_angle(sim.get_agent(0).get_state().rotation)
        positions.append(position)
        rotations.append(rotation)
        obs = sim.get_sensor_observations()
        rgb = np.concatenate([obs['panoramic_rgb_part_sensor_%d' % (i)][:, :, :3] for i in range(num_of_camera)], 1)
        semantic_array = np.concatenate([obs['panoramic_semantic_part_sensor_%d' % (i)] for i in range(num_of_camera)], 1)
        depth_array = np.concatenate([obs['panoramic_depth_part_sensor_%d' % (i)] for i in range(num_of_camera)], 1) / 10.
        bboxes, bbox_categories, bbox_ids, bbox_depth, bbox_pose = get_objects(semantic_array, mapping, depth_array, rgb.shape[1], rgb.shape[0])

        image_feat = get_image_feat(rgb, depth_array).cpu().detach().numpy()
        object_feat = []
        if len(bboxes) > 0:
            object_feat = get_object_feat(rgb, bboxes).cpu().detach().numpy()

        imggraph = update_image_graph(imggraph, objgraph, image_feat[0], object_feat, np.ones(len(bboxes)), bbox_categories, 
                                    position, rotation, time=t, done=False)

        if len(bboxes) > 0:
            objgraph = update_object_graph(imggraph, objgraph, object_feat, np.ones(len(bboxes)), bbox_categories, np.ones(len(bboxes)), 
                                        bbox_pose, time=t, done=False)

        topdownmap = draw_graph_on_map(topdownmap, imggraph, objgraph)
        topdownmap = draw_agent(topdownmap, cur_map_coord, cur_map_rot)
        rgb = draw_bbox(rgb, bboxes)
        topdownmaps.append(concat_vertical(rgb, topdownmap))
        cv2.imshow(f'vis', rgb[:, :, [2, 1, 0]])
        cv2.imshow(f'graph', topdownmap[::2, ::2, [2, 1, 0]])    
        t = t + 1    
    topdownmaps = np.stack(topdownmaps)

    fig = plt.figure(figsize=(10,10))
    plt.axis('off')
    plt.gca().set_axis_off()
    plt.subplots_adjust(top = 1, bottom = 0, right = 1, left = 0, hspace = 0, wspace = 0)
    plt.margins(0,0)

    im = plt.imshow(topdownmaps[0])
    plt.close() # this is required to not display the generated image
    anim = animation.FuncAnimation(fig, animate, init_func=init, frames=topdownmaps.shape[0], interval=50)
    HTML(anim.to_html5_video())
    writergif = animation.PillowWriter(fps=20) 
    anim.save(os.path.join(project_dir, "demo/tsgm_demo.gif"), writer=writergif)

[tayyabkhalil-313]

My kernel in the notebook crashes at
sim = habitat_sim.Simulator(cfg) I will check if your code runs through.

[Kenn3o3]

My kernel in the notebook crashes at sim = habitat_sim.Simulator(cfg) I will check if your code runs through.

Btw, my kernel also kept crashing last night, It took my a long while to figure out that it's my cv2 package causing the issue, and then I switched to python file.

[tayyabkhalil-313]

Thank you for the py file, running that indeed solves the kernel crashing issue. Now it successfully loads the scene but produces an error while trying to get the objects. Traceback (most recent call last): File "demo.py", line 670, in <module> bboxes, bbox_categories, bbox_ids, bbox_depth, bbox_pose = get_objects(semantic_array, mapping, depth_array, rgb.shape[1], rgb.shape[0]) File "demo.py", line 365, in get_objects max_key = np.max(np.array(list(mapping.keys()))) File "<__array_function__ internals>", line 200, in amax File "/home/tayyab/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 2820, in amax return _wrapreduction(a, np.maximum, 'max', axis, None, out, File "/home/tayyab/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction return ufunc.reduce(obj, axis, dtype, out, **passkwargs) ValueError: zero-size array to reduction operation maximum which has no identity Did you get a similar error?

[tayyabkhalil-313]

The reason for this error seems to be the empty mapping dict, which is a result of scene_objects = sim.semantic_scene.objects being empty. So, the simulator is unable to find any semantics in the scene. I have tried changing up the scene, but the same issue persists.

[tayyabkhalil-313]

@Kenn3o3 I have managed to run the demo notebook and the image encoder to work for the Allensville scan. I had to convert the original dataset via the 3DSceneGraph dataset as described here.

[Kenn3o3]

@Kenn3o3 I have managed to run the demo notebook and the image encoder to work for the Allensville scan. I had to convert the original dataset via the 3DSceneGraph dataset as described here.

Thanks!

[WarrenDream]

@Kenn3o3我已经成功运行演示笔记本和图像编码器，以便对 Allensville 进行扫描。我必须按照此处所述通过3DSceneGraph数据集转换原始数据集。 Can you take a look at the converted data directory structure and how the generated content is placed in the scene_data directory after I run the installation instructions