Closed swearos closed 1 month ago
Hi @swearos , thank you for your interest in SpatialBot.
(a) Names explained in Table 1. Sorry for the confusion in paper,
(b) RGBD evaluating: We prepare RGB and Depth maps, and feed them into model as 2 image inputs. SpatialBot, and Bunny implemented here, both support multi image inputs. You may want to refer to RGB/RGBD evaluation codes, where we provide RGB/RGBD evaluation for SpatialBench, MME and GQA.
(c) Why not report depth scores of GPT-4o: TLDR: it behaves badly. We are not sure, whether GPT-4o doesn't have the abality to do monocular depth estimation, or the prompt we use is not good. We are trying different prompts for GPT-4o, and will consider releasing depth scores for it in future versions. For fair comparasion, we do not report it for now.
(d) Can SpatialBot benefit from RGB-Depth, compared to RGB, in position, existence, ... ?
Hope it makes sense. Feel free to reach out if you have further questions.
Regards
Thank you for your work and your patient response!
I have one more question about the SpatialBench dataset: How do you build the QA data for depth without publishing the depth evaluation task?
https://huggingface.co/datasets/RussRobin/SpatialBench/tree/main
In SpatialBench HF, we release annotated QAs for high-level tasks (see paper Fig. 2).
Depth and proximity QAs are not released. They are in almost the same format with SpatialQA, which will be released soon. For depth, we ask the depth of points, and depth of objects. For proximity, we ask the model to compare the depth of two points, or two objects. It simply goes like: What is the depth value of point <0.55,0.73> / What is the depth value of object: a basketball?
I'll close this issue since no further discussion was raised for a week. Feel free to reopen it if you still have questions in our paper.
The new version of your arxiv pdf (July 16) has a major change in the chart that seems to introduce some ambiguous descriptions For example, in table 1: “LLM-RGB and LLM-RGBD are trained on RGB images only and tested with RGB and RGBD inputs,” But the training dataset of SpatialBot series models seems to contain RGBD data.
![image](https://github.com/user-attachments/assets/27dc5fbe-d59b-4685-9948-715bea4d86fc)
There are also questions to consult:
Look forward to your reply very much, thank you!