search

spla-tam / SplaTAM

SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM (CVPR 2024)
https://spla-tam.github.io/
BSD 3-Clause "New" or "Revised" License
1.59k stars 174 forks source link

Query on Info logged using Wandb #89

Closed lvmingzhe closed 5 months ago

lvmingzhe commented 8 months ago

Hello,

Thanks for your great work and I've come across several PSNR metrics that I would like to understand better. Specifically, I'm referring to Tracking/PSNR, Mapping/PSNR, and Final Stats/Average PSNR. Could you please clarify the differences between these metrics? I'm interested in how each is calculated.

Additionally, I've noticed that the Tracking step is showing a value of 0 in my results. Could you provide some insight into why this might be happening? I'm wondering if this is indicative of an issue with my data, a misunderstanding on my part regarding what the Tracking step represents, or if it's a potential bug in the software.

I've already reviewed the documentation available at previous issues but couldn't find the detailed explanations or troubleshooting guidance I was looking for.

Any guidance or references to more detailed resources would be greatly appreciated.

Thank you for your time and help.

From wandb summary log:

"Tracking/PSNR": 35.285186767578125, "Tracking/Depth RMSE": 0.007432922255247831, "Tracking/Depth L1": 0.007432922255247831, "Tracking/step": 0, "Tracking/Latest Pose Error": 0.0, "Tracking/Latest Relative Pose Error": 0.0, "Tracking/ATE RMSE": 0.0, "_timestamp": 1709986693.87413, "_runtime": 2236.5880041122437, "_step": 51530, "Per Iteration Mapping/Loss": 0.043003521859645844, "Per Iteration Mapping/Image Loss": 0.03194858878850937, "Per Iteration Mapping/Depth Loss": 0.011054933071136475, "Per Iteration Mapping/step": 17099, "Mapping/Number of Gaussians - Pruning": 6095185, "Mapping/step": 17100, "Mapping/PSNR": 16.289276123046875, "Mapping/Depth RMSE": 0.00043166676186956465, "Mapping/Depth L1": 0.00043166676186956465, "Per Iteration Tracking/Loss": 5655.00341796875, "Per Iteration Tracking/Image Loss": 1648.0791015625, "Per Iteration Tracking/Depth Loss": 4006.92431640625, "Per Iteration Tracking/step": 17039, "Mapping/Number of Gaussians": 6099206, "Final Stats/Average Tracking Iteration Time (ms)": 58.094412564111984, "Final Stats/Average Tracking Frame Time (s)": 3.4751132212187112, "Final Stats/Average Mapping Iteration Time (ms)": 69.2872304804841, "Final Stats/Average Mapping Frame Time (s)": 4.158240836126763, "Final Stats/step": 1, "Final Stats/Avg ATE RMSE": 0.02060262257340403, "Final Stats/Average PSNR": 24.738697052001953, "Final Stats/Average Depth RMSE": 0.013846458867192268, "Final Stats/Average Depth L1": 0.013846458867192268, "Final Stats/Average MS-SSIM": 0.903265655040741, "Final Stats/Average LPIPS": 0.13082163035869598,

Nik-V9 commented 8 months ago

Hi, you can better understand the logging by looking at the report_progress function and the following places of the code:

Basically this is the short explanation:

Also I would recommend looking at the plots in Wandb as opposed to these summary metrics.

lvmingzhe commented 8 months ago

Thank you very much for your detailed and patient response. I successfully found the code related to the entire wandb logging process and understood the general flow. I will follow your advice and focus more on the wandb curves instead of these final metrics. However, I have a small question that I need to confirm: it seems that there wasn't any code specifically written for adding final metrics to the wandb summary in your code. These results should be generated by the system automatically, right? Because I noticed in the summary document there is content like "Tracking/step": 0, but the tracking shouldn't be 0 steps, usually, it should be a much larger number. If you could confirm this small issue for me, I would be immensely grateful.

Nik-V9 commented 5 months ago

Hi, the final metrics are logged here: https://github.com/spla-tam/SplaTAM/blob/1a5d407c3a145f69691d49267411cb65e5d5c992/utils/eval_helpers.py#L595 https://github.com/spla-tam/SplaTAM/blob/1a5d407c3a145f69691d49267411cb65e5d5c992/scripts/splatam.py#L955

We use a dummy step number to log the final values. I'm not sure about the wandb summary steps, I generally don't look at them since we don't log them through the code but wandb generates it.