We introduce a novel approach to fine-grained cross-view geo-localization. Our method aligns a warped ground image with a corresponding GPS-tagged satellite image covering the same area using homography estimation. We first employ a differentiable spherical transform, adhering to geometric principles, to accurately align the perspective of the ground image with the satellite map. To address challenges such as occlusion, small overlapping range, and seasonal variations, we propose a robust correlation-aware homography estimator to align similar parts of the transformed ground image with the satellite image. Our method achieves sub-pixel resolution and meter-level GPS accuracy by mapping the center point of the transformed ground image to the satellite image using a homography matrix and determining the orientation of the ground camera using a point above the central axis. Operating at a speed of 30 FPS, our method outperforms state-of-the-art techniques, reducing the mean metric localization error by 21.3% and 32.4% in same-area and cross-area generalization tasks on the VIGOR benchmark, respectively, and by 34.4% on the KITTI benchmark in same-area evaluation.
HC-Net is online! Try it at this url.
Please use the demo script to deploy the demo locally for testing.
You can test our model using the data from the 'same_area_balanced_test.txt' split of the VIGOR dataset, or by providing your own Panorama image along with its corresponding Satellite image.
We train and test our codes under the following environment:
To get started, follow these steps:
git clone https://github.com/xlwangDev/HC-Net.git
cd HC-Net
conda create -n hcnet python=3.8 -y
conda activate hcnet
pip install -r requirements.txt
sh train.sh
To evaluate the HC-Net model, follow these steps:
chmod +x val.sh
# Usage: val.sh [same|cross]
# For same-area in VIGOR
./val.sh same 0
# For cross-area in VIGOR
./val.sh cross 0
python demo_gradio.py
We propose the use of Mercator projection to directly compute the pixel coordinates of ground images on specified satellite images using the GPS information provided in the dataset. You can find the specific code at Mercator.py.
To use our corrected label, you can add the following content to the __getitem__
method of the VIGORDataset
class in datasets.py
file in the CCVPE project:
from Mercator import *
pano_gps = np.array(self.grd_list[idx][:-5].split(',')[-2:]).astype(float)
pano_gps = torch.from_numpy(pano_gps).unsqueeze(0)
sat_gps = np.array(self.sat_list[self.label[idx][pos_index]][:-4].split('_')[-2:]).astype(float)
sat_gps = torch.from_numpy(sat_gps).unsqueeze(0)
zoom = 20
y = get_pixel_tensor(sat_gps[:,0], sat_gps[:,1], pano_gps[:,0],pano_gps[:,1], zoom)
col_offset_, row_offset_ = y[0], y[1]
width_raw, height_raw = sat.size
col_offset, row_offset = width_raw/2 -col_offset_.item(), row_offset_.item() - height_raw/2
We have released the code corresponding to section A.2 in the paper's Supplementary, along with an online testing platform .
Compared to traditional Inverse Perspective Mapping (IPM), our approach does not require calibration of camera parameters. Instead, it allows for manual tuning to achieve an acceptable BEV projection result.
You can use Hugging Face Spaces for online testing, or run our code locally. Online testing utilizes CPU for computation, which is slower. If you run it locally with a GPU, the projection process takes less than 10ms.
python demo_gradio_kitti.py
Our projection process is implemented entirely in PyTorch, which means our projection method is differentiable and can be directly deployed in any network for gradient propagation.
If you find our work helpful, please cite:
@article{wang2024fine,
title={Fine-Grained Cross-View Geo-Localization Using a Correlation-Aware Homography Estimator},
author={Wang, Xiaolong and Xu, Runsen and Cui, Zhuofan and Wan, Zeyu and Zhang, Yu},
journal={Advances in Neural Information Processing Systems},
volume={36},
year={2024}
}