search

zju3dv / EfficientLoFTR

Apache License 2.0
635 stars 48 forks source link

我修改的jupyterlab #5

Closed jxhoh closed 8 months ago

jxhoh commented 8 months ago

import os os.chdir("..") from copy import deepcopy

import torch import cv2 import numpy as np import matplotlib.cm as cm from src.utils.plotting import make_matching_figure from src.utils.plotting import make_matching_figure_2 print(torch.cuda.is_available()) from src.loftr import LoFTR,default_cfg _default_cfg = deepcopy(default_cfg) _default_cfg['coarse']['temp_bug_fix'] = True # set to False when using the old ckpt _default_cfg['coarse']['npe'] = [832, 832, 832, 832] print(_default_cfg)

matcher = LoFTR(config=_default_cfg) matcher.load_state_dict(torch.load('D:\jx\ai\featue_match\ELOFTR\EfficientLoFTR-main\eloftr_outdoor.ckpt')['state_dict']) matcher.eval().cuda()

Load example images

img0_pth = "D:\jx\ai\imageRdataset\models-master\models-master\research\delf\delf\python\examples\data\oxford5k_images\Snipaste_2024-03-14_17-34-54_r.jpg" img1_pth = "D:\jx\ai\imageRdataset\models-master\models-master\research\delf\delf\python\examples\data\oxford5k_images\Snipaste_2024-03-14_17-31-37.jpg"

img0_pth = "D:\jx\data\040-1-014231-514-004490.jpg"

img1_pth = "D:\jx\data\040-2-014231-514-004490.jpg"

img0_raw = cv2.imread(img0_pth, cv2.IMREAD_GRAYSCALE) img1_raw = cv2.imread(img1_pth, cv2.IMREAD_GRAYSCALE) img0_raw = cv2.resize(img0_raw, (640, 480)) img1_raw = cv2.resize(img1_raw, (640, 480))

img0 = torch.from_numpy(img0_raw)[None][None].cuda() / 255. img1 = torch.from_numpy(img1_raw)[None][None].cuda() / 255. batch = {'image0': img0, 'image1': img1}

Inference with LoFTR and get prediction

with torch.no_grad(): matcher(batch) mkpts0 = batch['mkpts0_f'].cpu().numpy() mkpts1 = batch['mkpts1_f'].cpu().numpy() mconf = batch['mconf'].cpu().numpy()

Draw

color = cm.jet(mconf) text = [ 'LoFTR', 'Matches: {}'.format(len(mkpts0)), ] fig = make_matching_figure_2(img0_raw, img1_raw, mkpts0, mkpts1, color, text=text,path="D:\jx\ai\imageRdataset\models-master\models-master\research\delf\delf\python\examples\data\oxford5k_images\result_1.jpg")

配置文件的内容我就这样写了 from yacs.config import CfgNode as CN

def lower_config(yacs_cfg): if not isinstance(yacs_cfg, CN): return yacs_cfg return {k.lower(): lower_config(v) for k, v in yacs_cfg.items()}

############## ↓ LoFTR Pipeline ↓ ############## _CN = CN() _CN.BACKBONE_TYPE = 'RepVGG' _CN.ALIGN_CORNER = False _CN.RESOLUTION = (8, 1) _CN.FINE_WINDOW_SIZE = 8 # window_size in fine_level, must be even _CN.FP16 = False _CN.REPLACE_NAN = False _CN.EVAL_TIMES = 1

1. LoFTR-backbone (local feature CNN) config

_CN.BACKBONE = CN() _CN.BACKBONE.BLOCK_DIMS = [64, 128, 256] # s1, s2, s3

2. LoFTR-coarse module config

_CN.COARSE = CN() _CN.COARSE.D_MODEL = 256 _CN.COARSE.D_FFN = 256 _CN.COARSE.NHEAD = 8 _CN.COARSE.LAYER_NAMES = ['self', 'cross'] * 4 _CN.COARSE.AGG_SIZE0 = 4 _CN.COARSE.AGG_SIZE1 = 4 _CN.COARSE.NO_FLASH = False _CN.COARSE.ROPE = True _CN.COARSE.NPE = None

3. Coarse-Matching config

_CN.MATCH_COARSE = CN() _CN.MATCH_COARSE.THR = 0.1 _CN.MATCH_COARSE.BORDER_RM = 2 _CN.MATCH_COARSE.DSMAX_TEMPERATURE = 0.1 _CN.MATCH_COARSE.TRAIN_COARSE_PERCENT = 0.2 # training tricks: save GPU memory _CN.MATCH_COARSE.TRAIN_PAD_NUM_GT_MIN = 200 # training tricks: avoid DDP deadlock _CN.MATCH_COARSE.SPARSE_SPVS = True _CN.MATCH_COARSE.SKIP_SOFTMAX = False _CN.MATCH_COARSE.FP16MATMUL = False

4. Fine-Matching config

_CN.MATCH_FINE = CN() _CN.MATCH_FINE.SPARSE_SPVS = True _CN.MATCH_FINE.LOCAL_REGRESS_TEMPERATURE = 1.0 _CN.MATCH_FINE.LOCAL_REGRESS_SLICEDIM = 8

5. LoFTR Losses

-- # coarse-level

_CN.LOSS = CN() _CN.LOSS.COARSE_TYPE = 'focal' # ['focal', 'cross_entropy'] _CN.LOSS.COARSE_WEIGHT = 1.0 _CN.LOSS.COARSE_SIGMOID_WEIGHT = 1.0 _CN.LOSS.LOCAL_WEIGHT = 0.5 _CN.LOSS.COARSE_OVERLAP_WEIGHT = False _CN.LOSS.FINE_OVERLAP_WEIGHT = False _CN.LOSS.FINE_OVERLAP_WEIGHT2 = False

-- - -- # focal loss (coarse)

_CN.LOSS.FOCAL_ALPHA = 0.25 _CN.LOSS.FOCAL_GAMMA = 2.0 _CN.LOSS.POS_WEIGHT = 1.0 _CN.LOSS.NEG_WEIGHT = 1.0

-- # fine-level

_CN.LOSS.FINE_TYPE = 'l2_with_std' # ['l2_with_std', 'l2'] _CN.LOSS.FINE_WEIGHT = 1.0 _CN.LOSS.FINE_CORRECT_THR = 1.0 # for filtering valid fine-level gts (some gt matches might fall out of the fine-level window)

default_cfg = lower_config(_CN) 不知道有没有问题 我jupyter跑出来的结果倒是没问题，就是有时候点太多了 精度感觉没有那么高

tan-tao11 commented 8 months ago

感谢！点确实有点多，请问有没有筛选的方法，去除一些精度比较低的点对？

jxhoh commented 8 months ago

感谢！点确实有点多，请问有没有筛选的方法，去除一些精度比较低的点对？

可以在程序里面调那个阈值，有个配置文件中的thr，我跟了一遍代码就出来了

wyf2020 commented 8 months ago

抱歉没有及时回复，我们已经更新一个jupyter notebook demo，上面你提供的jupyterlab的模型调用是对的，但是有几个需要修改的地方 (增加重参数化，THR设为0.2，MATCH_FINE.LOCAL_REGRESS_TEMPERATURE设为10.0)，建议直接使用我们更新的demo！

关于点太多的问题，这是一个非常好的发现！事实上，我们推荐full和opt. model分别使用thr=0.2和thr=25，你也可以根据实际情况自己调节。复现脚本给的(full thr=0.1, opt. thr=20)只是为了准确复现论文里AUC数值，我们在这里列出两组thr在MegaDepth和ScanNet测试集上的精度，以便更好理解和使用我们的模型。另外，你也可以看到我们的project page里的可视化demo的左下角，所有ours (full) model都是使用thr=0.2作为阈值的。

thr. (full/opt.) Megadepth AUC[5/10/20] ScanNet AUC[5/10/20]
Reproduce paper (0.1/20) Full(56.4/72.2/83.5) / Opt.(55.4/71.4/82.9) Full(19.2/37.0/53.6) / Opt.(17.4/34.4/51.2)
Recommended (0.2/25) Full(55.6/71.7/83.2) / Opt.(55.5/71.6/83.0) Full(19.5/37.5/54.1) / Opt.(18.5/35.8/52.2)
jxhoh commented 8 months ago

抱歉没有及时回复，我们已经更新一个jupyter notebook demo，上面你提供的jupyterlab的模型调用是对的，但是有几个需要修改的地方 (增加重参数化，THR设为0.2，MATCH_FINE.LOCAL_REGRESS_TEMPERATURE设为10.0)，建议直接使用我们更新的demo！

关于点太多的问题，这是一个非常好的发现！事实上，我们推荐full和opt. model分别使用thr=0.2和thr=25，你也可以根据实际情况自己调节。复现脚本给的(full thr=0.1, opt. thr=20)只是为了准确复现论文里AUC数值，我们在这里列出两组thr在MegaDepth和ScanNet测试集上的精度，以便更好理解和使用我们的模型。另外，你也可以看到我们的project page里的可视化demo的左下角，所有ours (full) model都是使用thr=0.2作为阈值的。

thr. (full/opt.) Megadepth AUC[5/10/20] ScanNet AUC[5/10/20] Reproduce paper (0.1/20) Full(56.4/72.2/83.5) / Opt.(55.4/71.4/82.9) Full(19.2/37.0/53.6) / Opt.(17.4/34.4/51.2) Recommended (0.2/25) Full(55.6/71.7/83.2) / Opt.(55.5/71.6/83.0) Full(19.5/37.5/54.1) / Opt.(18.5/35.8/52.2)

okok，感谢答复