search

Parskatt / RoMa

[CVPR 2024] RoMa: Robust Dense Feature Matching; RoMa is the robust dense feature matcher capable of estimating pixel-dense warps and reliable certainties for almost any image pair.
https://parskatt.github.io/RoMa/
MIT License
434 stars 33 forks source link

Training code in `roma_indoor.py` seems to use non-distributed sampler with DDP #32

Closed justachetan closed 2 months ago

justachetan commented 2 months ago

Hi,

I was going through the training code in experiments/roma_indoor.py and it seems that you have used a non-distributed sampler (WeightedRandomSampler) instead of DistributedSampler. I believe this means that the entire data will be replicated and passed to each model replica instead of shards of the data. Just wanted to confirm this and ask if this is this is intentional?

Thanks!

Parskatt commented 2 months ago

Yes all data goes to all. Its not really intentional. Distributed sampler should be used but I think the effect would be minor. I guess if youre sending entire scannet to each gpu it will be annoyingly heavy. Btw, I just used every 10th frame of scannet iirc (it was a while since I used the code)

justachetan commented 2 months ago

Thanks! I am not sure what happens when a non-distributed sampler works with distributed training. I am assuming each replica of the dataloader gets a different seed and the order of samples is different across devices?

Parskatt commented 2 months ago

I assume so. I never set any seeds, and I havent observed issues relating to repeated samples.

justachetan commented 2 months ago

Thank you!