About Loading Nr3D annotations.

[RunsenXu]

Dear Authors,

Thanks for your great work!

I am writing to ask why we should filter some of the samples in Nr3D and we only test on samples with correct_guess == True. Is this a typical protocol for this benchmark.

    def load_nr3d_annos(self):
        """Load annotations of nr3d."""
        split = self.split
        if split == 'val':
            split = 'test'
        with open('data/meta_data/nr3d_%s_scans.txt' % split) as f:
            scan_ids = set(eval(f.read()))
        with open(self.data_path + 'nr3d_pred_spans.json', 'r') as f:
            pred_spans = json.load(f)
        with open(self.data_path + 'refer_it_3d/nr3d.csv') as f:
            csv_reader = csv.reader(f)
            headers = next(csv_reader)
            headers = {header: h for h, header in enumerate(headers)}
            annos = [
                {
                    'scan_id': line[headers['scan_id']],
                    'target_id': int(line[headers['target_id']]),
                    'target': line[headers['instance_type']],
                    'utterance': line[headers['utterance']],
                    'anchor_ids': [],
                    'anchors': [],
                    'dataset': 'nr3d',
                    'pred_pos_map': pred_spans[i]['span'],  # predicted span
                    'span_utterance': pred_spans[i]['utterance']  # for assert
                }
                for i, line in enumerate(csv_reader)
                if line[headers['scan_id']] in scan_ids
                and
                str(line[headers['mentions_target_class']]).lower() == 'true'
                and
                (
                    str(line[headers['correct_guess']]).lower() == 'true'
                    or split != 'test'
                )
            ]

[RunsenXu]

In the original Referit3D codebase, I can find that they filter those with false values for "mentions_target_class".

https://github.com/referit3d/referit3d/blob/725a5d31b8ff3647b6fb5342304093e9568d1341/referit3d/in_out/neural_net_oriented.py#L64 But I can't find the relevant part of 'correct_guess'.

[ayushjain1144]

from the official website: https://referit3d.github.io/benchmarks.html

[RunsenXu]

I see. Thank you very much!