About RobotCar v2 results

[RuotongWANG]

Hi, I really appreciate for your work. I'm trying to reproduce your result on RobotCar seasons v2 dataset and encountered some problems. Which dataset split is used in the Tables 1 and 2 of the main paper, test or train? Besides, are these results integrate ALL conditions including day and night?

I noticed that the Suppl. Table 1. reports results for each condition obtained from the training split. But when I tried to summary these results in Suppl. Table 1. using statistics on the train query set, I failed to obtain the results in the Tables 1 of the main paper. So I'm a bit confused about the dataset settings.

I am always looking forward to your kind response.

Best regards,

[StephenHausler]

Hi RuotongWANG,

Tables 1 and 2 of the main paper use the test split. You will be unable to directly replicate our results on this dataset - the numbers in our paper are generated by the challenge submission server at: https://www.visuallocalization.net/ The ground truth data for the test split of RobotCar seasons v2 has not been publicly released, evaluation is only possible through submission to the challenge server. You will need to make a submission of your own in order to reproduce our results.

Yes, Tables 1 and 2 integrate all conditions including day and night.

Yes, Suppl. Table 1. reports results from each condition, using the training split. Ground truth data has been released for the training split, which allowed us to produce this detailed breakdown of the performance by condition. You will be unable to match the Suppl. Table 1 results to the main paper results, since these are different splits.

Hope this helps.

[RuotongWANG]

Thanks for your response. Have a nice day :)

[HeartbreakSurvivor]

Hi RuotongWANG,

Tables 1 and 2 of the main paper use the test split. You will be unable to directly replicate our results on this dataset - the numbers in our paper are generated by the challenge submission server at: https://www.visuallocalization.net/ The ground truth data for the test split of RobotCar seasons v2 has not been publicly released, evaluation is only possible through submission to the challenge server. You will need to make a submission of your own in order to reproduce our results.

Yes, Tables 1 and 2 integrate all conditions including day and night.

Yes, Suppl. Table 1. reports results from each condition, using the training split. Ground truth data has been released for the training split, which allowed us to produce this detailed breakdown of the performance by condition. You will be unable to match the Suppl. Table 1 results to the main paper results, since these are different splits.

Hope this helps.

Hi, StephenHausler. I'm trying to reproduce your result on RobotCar seasons v2 dataset , I download the robotcar v2 dataset from here, the robotcar_v2_train.txt contains the names and poses of the training images and has the following format: image_name R00 R01 R02 C0 R10 R11 R12 C1 R20 R21 R22 C2 0 0 0 1 with one line per image.

In order to get result from https://www.visuallocalization.net/, I need make a submission to the challenge server. But how can I convert this to a submit format as mentioned in here. like this:

rear/1417178853389469.jpg 0.655230866337965 -0.536609290501044 -0.340346549997501 -0.408516459863989 24.094399999999997 24.855200000000004 -120.244000000000028

Or could you please provide robotcar v2 db pose data file?

[StephenHausler]

Hi @HeartbreakSurvivor

To run the RobotCar Seasons v2 challenge (query) set on Patch-NetVLAD, you need to run Patch-NetVLAD to produce a pairs file then use Kapture (https://github.com/naver/kapture) to convert the pairs file into a LTVL challenge file. The LTVL challenge file is the format needed to submit to the challenge server.

I've copied and pasted some example code which converts the pairs file to the challenge format, using Kapture. The pairs file is created by default when you run Patch-NetVLAD. You also need to download the RobotCar Seasons v2 dataset in the Kapture format (or convert it to Kapture). Note that many previous works using this dataset split the dataset into multiple submaps - we don't do this, and evaluate all images as one large image retrieval problem.

import argparse
import os
from collections import defaultdict
import pathlib

import kapture
import kapture.io.csv as csv

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='use pairsfiles and kaptures for pose estimation using image retrieval')
    parser.add_argument('-p', '--pairsfile-path', type=str, required=True, help='path to pairsfile')
    parser.add_argument('-m', '--mapping-path', type=str, required=True, help='path to mapping kapture')
    parser.add_argument('-o', '--output-path', type=str, required=True, help='path to output LTVL challenge file')
    parser.add_argument('-d', '--decreasing', action='store_true', help='set if descending scores indicate a better match')
    parser.add_argument('-i', '--inverse', action='store_true', help='invert poses before recording them down in output file')
    args = parser.parse_args()

    kdata_mapping = csv.kapture_from_dir(args.mapping_path)
    if kdata_mapping.rigs:
        kapture.rigs_remove_inplace(kdata_mapping)

    with open(args.pairsfile_path, 'r') as f:
        image_pairs = csv.table_from_file(f)

    query_lookup = defaultdict(list)    

    for query, mapping, score in image_pairs:
        query_lookup[query] += [(mapping, score)]

    # locate best match using pairsfile
    best_match_pairs = []
    for query, retrieved_mapping in query_lookup.items():
        if args.decreasing:
            best_match = min(retrieved_mapping, key=lambda x: x[1])[0]
        else:
            best_match = max(retrieved_mapping, key=lambda x: x[1])[0]
        best_match_pairs += [(query, best_match)]

    # recover pose from best match
    fname_to_pose_lookup = {}
    for ts, cam, fname in kapture.flatten(kdata_mapping.records_camera):
        fname_to_pose_lookup[fname] = kdata_mapping.trajectories[ts, cam]
    image_poses = {query: fname_to_pose_lookup[mapping] for query, mapping in best_match_pairs}

    # LTVT2020
    p = pathlib.Path(args.output_path)
    os.makedirs(str(p.parent.resolve()), exist_ok=True)
    with open(args.output_path, 'wt') as f:
        for image_filename, pose in image_poses.items():
            if args.inverse:
                pose = pose.inverse()
            line = [image_filename] + pose.r_raw + pose.t_raw
            line = ' '.join(str(v) for v in line) + '\n'
            f.write(line)`