search

dcharatan / flowmap

[3DV 2025] Code for "FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent" by Cameron Smith*, David Charatan*, Ayush Tewari, and Vincent Sitzmann
https://cameronosmith.github.io/flowmap/
MIT License
893 stars 87 forks source link

Randomness in Results #42

Closed Nik-V9 closed 4 months ago

Nik-V9 commented 5 months ago

Hey @dcharatan, Thanks for the nice code!

I've been running the code through the overfit.py file on my own data. Even though I seed everything, the code produces different results each time (sometimes the performance delta is large).

I was wondering if there are any elements of the code which have randomness that can't be seeded? What is the best way to seed the code for benchmarking? For the paper evaluations, how do you go about this issue?

I tried setting the PyTorch Lightning Trainer to determinisitic but there were a few operations like 2D grid sampling which didn't have a determinisitc implementation.

I'm attaching snippets of my evaluation here across 6 back-to-back runs with the same command and seed:

image image image image image image
dcharatan commented 4 months ago

While working on the code, we weren't careful to ensure that all randomness comes from the initial seed. You could go through the code to make sure all randomness is seeded properly, but it's probably best to just run the code several times and report a mean and standard deviation in the results for benchmarking purposes.