Question about map input in "oracle" setting and "oracle-ego" setting

[JeremyLinky]

Hi! Thanks for the wonderful work first! I took a look at your codes and have some confuses about "oracle" setting and "oracle-ego" setting. First, could you tell me that are "oracle-ego" and “OracleEgoMap” the same thing? Because I noticed that in your codes, you didn't utilize the occupancy channel of map when in "oracle-ego" setting, while you said in paper that OracleEgoMap also had an occupancy channel.

Second, the object predicted by ObjRecogMap is only the object contained in the current rgb observation, which is different from that of OracleEgoMap, right? There is only goal object in the semantic channel of OracleEgoMap, and the semantic value is only assigned near the goal object‘s position, am I right?

Moreover, is the embedding of the object classified by the ObjRecogMap placed at the current agent's position, or the actual position of the object? If it is the actual position corresponding to the object, how do you determine the actual position corresponding to the object? if not , then if the robot is standing at the same position on the second time and turning in different direction, the rgb value it sees is also different, and the predicted object is also different, will the value of the robot's location be changed at this time?

Thanks for your patience! Looking forward to your reply!

[saimwani]

Yes, oracle-ego in the code refers to OracleEgoMap as described in the README. When oracle-ego is passed as an argument, the occupancy channel is indeed used.

Second, the object predicted by ObjRecogMap is only the object contained in the current rgb observation, which is different from that of OracleEgoMap, right? There is only goal object in the semantic channel of OracleEgoMap, and the semantic value is only assigned near the goal object‘s position, am I right?

Yes and yes.

Moreover, is the embedding of the object classified by the ObjRecogMap placed at the current agent's position, or the actual position of the object?

The embedding of the object classified is placed at the current agent position, not the actual position of the object.

if not , then if the robot is standing at the same position on the second time and turning in different direction, the rgb value it sees is also different, and the predicted object is also different, will the value of the robot's location be changed at this time?

Yes, the value is over-written. The supplementary section of our paper describes this procedure.

[JeremyLinky]

Thanks for your reply!! I still couldn't find where occupancy channel is used when oracle-ego is passed as an argument, but I find the following code:

 if self.agent_type != "no-map":
            global_map_embedding = []
            global_map = observations['semMap']
            if self.agent_type == "oracle":
                global_map_embedding.append(self.occupancy_embedding(global_map[:, :, :, 0].type(torch.LongTensor).to(self.device).view(-1)).view(bs, 50, 50 , -1))
            global_map_embedding.append(self.object_embedding(global_map[:, :, :, 1].type(torch.LongTensor).to(self.device).view(-1)).view(bs, 50, 50, -1))
            global_map_embedding = torch.cat(global_map_embedding, dim=3)
            map_embed = self.map_encoder(global_map_embedding)
            x = [map_embed] + x

Does it show that when self.agent_type == "oracle-ego", there is no self.occupancy_embedding? Besides, it seems that there are no models of "proj-neural" and "obj-recog", is it true? Looking for your reply!