Training with own dataset on both extractor and matcher

[XiujinLiu]

Hi I'm trying to try the LightGlue and SuperPoint with my own dataset, I have several questions:

1. is there any way to train both at the same time? or I need to train SuperPoint first get the finetuned weight and then bring the weight to lightGlue to train LightGlue?
2. Is there any format example for the dataset and the label (groundtruth) for both LightGlue and SuperPoint?
3. I didn't find the place where I can load the training data
4. the https://github.com/rpautrat/SuperPoint?tab=readme-ov-file shows the training file using tensorflow, but I find out there is a pytorch version of model with out any loss and backward, is that means we cannot use pytorch to train the model , but use pytorch to predict the interest point and descriptor? Thanks in advance!

[Phil26AT]

Hi @XiujinLiu

1. It might be possible to train them at the same time, although its likely easier to train them separately (or at least pre-train the detector, and train descriptor+matcher together).
2. All you need is the images (RGB), the depth map (same shape as images), and intrinsics+extrinsics. We support loading from COLMAP formats (both for poses and cameras). pycolmap might be helpful here.
3. The dataloader for MegaDepth is here.
4. The superpoint-open model was trained in tensorflow back in the days. For convenience we ported the weights to pytorch, but not yet the training code. However, you could probably port the loss function to pytorch, and then train it with glue-factory.

[pxy522]

你可以试试scannet？可能需要自己换种形式训练