search

cshizhe / onav_rim

MIT License
32 stars 1 forks source link

Poor Model Performance on HM3D Dataset in Cross-Domain Setting #7

Closed yusirhhh closed 5 months ago

yusirhhh commented 5 months ago

Upon transferring the model trained on the MP3D (Matterport3D) dataset for evaluation on the HM3D dataset, I have noticed a substantial decline in its performance under the cross-domain setting. However, I am uncertain about the root cause of this issue. Could you please provide any relevant configuration files or offer suggestions to pinpoint the problem?

Here's a comparison of the outcomes:

Evaluation results on the HM3D dataset:

2024-04-03 20:58:52,680 Object categories: 28
2024-04-03 20:58:52,680 step input size: 512
2024-04-03 23:11:15,812 Average episode reward: 2.9890
2024-04-03 23:11:15,812 Average episode distance_to_goal: 5.2411
2024-04-03 23:11:15,812 Average episode success: 0.3100
2024-04-03 23:11:15,812 Average episode spl: 0.1413
2024-04-03 23:11:15,812 Average episode softspl: 0.2381
2024-04-03 23:11:15,812 Average episode collisions.count: 10.4160

In comparison, the model performed notably better within the MP3D dataset:

2024-03-06 06:43:06,888 Object categories: 28
2024-03-06 06:43:06,888 step input size: 512
2024-03-06 15:32:29,139 Average episode reward: 4.8421
2024-03-06 15:32:29,140 Average episode distance_to_goal: 4.0260
2024-03-06 15:32:29,140 Average episode success: 0.4998
2024-03-06 15:32:29,140 Average episode spl: 0.1613
2024-03-06 15:32:29,140 Average episode softspl: 0.2159
2024-03-06 15:32:29,140 Average episode collisions.count: 15.7809
cshizhe commented 5 months ago

Hi, it seems that you evaluated on 28 categories on HM3D. But the standard testing setup in HM3D only performs evaluation on 6 object goal categories: chair, couch, potted plant, bed, toilet and tv.

yusirhhh commented 5 months ago

Thank you for your reply. I've identified and corrected the issue: the object goal indices differed between the MP3D and HM3D dataset.

sxyxs commented 1 week ago

Hi @yusirhhh, I noticed that the model.net.obj_categories_embedding.weight is manually set by a fixed size of 28 here with the nn.Embedding shape. How did you adjust the nn.Embedding shape when loading the pre-trained model for evaluation? Did you just load the first 6 rows for HM3D? Could you share how you adjusted the MP3D setting to work with HM3D? Thanks!