search

eigenvivek / DiffPose

[CVPR 2024] Intraoperative 2D/3D registration via differentiable X-ray rendering
http://vivekg.dev/DiffPose/
MIT License
118 stars 14 forks source link

A little question about PnP Regularizer #26

Closed yewentao2000 closed 1 month ago

yewentao2000 commented 5 months ago

Dear eigenvivek: Thank you very much for your excellent paper. I wonder that the PnP Regularizer method in your paper use this Xreg code (https://github.com/rg2/Regi2D3D-IPCAI2020)? When I use Python to reproduce on DeepFluoro, I can't achieve particularly fast speed and accuracy (about 10s and mTRE=2.8mm). Is this because I am using a full-image, full-pixel intensity-based CMAES optimization? Do you think my situation is reasonable? Thank you for your review and prompt reply!

eigenvivek commented 5 months ago

Hi @yewentao2000 , yes the PnP regularizer comes from the Rigi2D3D repo. I ran it using instructions on their wiki, so you should just be able to follow that.

Full images will take longer to render, so that’s why it’s slower. Also, I’ve never used CMAES before. Is it running on the GPU? If not, it will be slow. Additionally, I’ve found that mTRE is very sensitive to the hyper parameters you use. Make sure to tune the CMAES hyper parameters.

yewentao2000 commented 5 months ago

Thank you for your reply. Yes, you used the Adam optimizer directly in your work. I saw that CMAES was used in the Rigid2D3D work, and it used some pelvic segmentation to participate in the optimization, so I am curious about the effect of full image registration without preprocessing. Maybe it is because the interference of full image information makes mTRE lower than reported?

eigenvivek commented 1 month ago

I'm not sure about that. I would expect using a higher-resolution image would get you better registration accuracy. That is unless your initial pose estimate is too far from the ground truth, in which case, higher-res images would probably get stuck in a local min.