search

Rishit-dagli / NeRF-US

Official code for NeRF-US: Removing Ultrasound Imaging Artifacts from Neural Radiance Fields in the Wild
https://rishitdagli.com/nerf-us
Apache License 2.0
9 stars 0 forks source link

Inquiry Regarding COLMAP for Determining Ultrasound Image Positions #1

Closed cshennju closed 2 months ago

cshennju commented 2 months ago

Thank you so much for your amazing work!

I have a question regarding the use of COLMAP to determine the position of different ultrasound images. Specifically, each ultrasound image represents a cross-section of a three-dimensional object. Could you please provide some guidance or share any resources on how to use COLMAP to determine the relative positions of these multiple cross-sections?

Thank you in advance for your assistance.

Rishit-dagli commented 2 months ago

thanks for pointing this out, i should have clarified this. instead of treating this like a 3d-object, for sfm we treat the images as an open scene and then use colamp. for running colmap, i use this script: https://github.com/NVlabs/instant-ngp/blob/master/scripts/colmap2nerf.py.

cshennju commented 2 months ago

Thank you for the clarification. However, I still have a question: since the intersection between two ultrasound images is minimal (for example, if the two cross-sections are perpendicular to each other, the intersection could be just a single line), how can COLMAP accurately calculate their relative positions? Given that the feature points between the two images are very few, wouldn't that impact COLMAP's accuracy?

Rishit-dagli commented 2 months ago

i completely agree with you, this is quite a bit of a problem especially with casual images.

  1. this was also a problem while building the 10 datasets for this work (https://huggingface.co/datasets/rishitdagli/us-in-the-wild) which have all been made by running colmap on a video (using the script with instant ngp). while making the datasets, i just experimented around and found these colmap parameters to work best:
    • aabb scale: 128
    • sample at 30 FPS
    • don't use sequential matching (try to match every image)
    • maybe this was just something with the way i installed colmap or my environment but i found the sift gpu implementation to not work well and did not use that
  2. i agree this is a problem in general with most kinds of nerfs since they require doing colmap first. i know there have been quite a few colmap-free methods that solve exactly this (+ a few other) problems with colmap. while i haven't experimented with them right now, i do think it would be interesting to do so and could help get around this problem.