search

eigenvivek / DiffPose

[CVPR 2024] Intraoperative 2D/3D registration via differentiable X-ray rendering
http://vivekg.dev/DiffPose/
MIT License
118 stars 14 forks source link

Questions about the optimal initial point for registration #6

Closed khh656 closed 8 months ago

khh656 commented 9 months ago

I have found better initial points in traditional rigid registration before, but I want to migrate to DiffDRR and find that it is not ideal（I used the astra library to generate DRR (https://github.com/astra-toolbox/astra-toolbox), which I set based on the actual projection conditions of the X-ray image）. So if I don't use my previous better initial points, will it affect the subsequent registration results? This question is bugging me right now, if you could answer it I would really appreciate it！

eigenvivek commented 9 months ago

Hi, thanks for your question. Just note that DiffPose (this repo) finds the initial point with a neural network, whereas you seem to want to set it manually. Manual initialization is more in line with DiffDRR.

In DiffDRR, you can still set the initial point. This would be the first view that you see in the registration optimization process (see this section of the 2D/3D registration tutorial).

How are you getting the "actual projection conditions of the X-ray image"? Is this from the DICOM header? If yes, you'll just need to translate that pose into DiffDRR's convention, and then you're good to go!

khh656 commented 9 months ago

I tried it on DiffDRR and it wasn't ideal, but thanks for your answer, I'm happy with it.

You said that an initial point can be obtained through neural network , but I still have questions, that is, if I use my own set of CT data for model training, and then use a set of matching X-ray images to input into the model, this set of -Due to equipment reasons, the X-ray image will have a relatively large gap in position from the DRR map generated using the DiffDRR algorithm (not only is there an offset on the two-dimensional plane, but there is also a gap in the scaling ratio), even so, the model can still Make a rough match between them? (And then fine-tune it on this basis to achieve a more accurate registration)，Thanks again for your answer.

eigenvivek commented 9 months ago

Just want to make sure I understand your question - when you say there is "an offset on the two-dimensional plane" and "a gap in the scaling ratio", are you talking about the principal point and the focal length? DiffDRR assumes that you have ground truth values for those parameters. If your X-rays are in DICOM, those parameters are stored in the header and you can extract them.

If you render a DRR with DiffDRR from the true extrinsic camera pose and use the true intrinsic parameters (principal point and focal length), the rendered X-ray should look very close to the real X-ray (of course, the real X-ray will be more detailed and realistic looking).

So just to clarify, the networks in DiffPose only predict the extrinsic parameters. There's no reason that they can't be trained to also predict your intrinsic parameters. That just wasn't done in the current work, but would be a great extension.

If you don't have good ground truth values for the intrinsic parameters, you can also estimate them with gradient descent in DiffDRR. In the latest version, I made the renderer differentiable w.r.t. these intrinsic parameters.

Did I understand/answer your original question or did I miss something?

khh656 commented 9 months ago

Thank you very much for your reply. Yes, your reply is very good (it eliminates a lot of doubts that troubled me). I also performed registration according to your suggestions (respectively for the X-Ray image of AP and the X-Ray image of LAT). ), I found that if I perform posture regression on translation and rotation, the effect is not very good (mainly due to the impact of rotation), but if I only perform posture regression on translation, the effect can be achieved. What should I do? On the premise of performing attitude regression on translation and rotation, can better results be achieved? Another more important issue is that I want to combine AP and LAT images (they are orthogonal in space, but the Source-to-detector of AP and LAT are different) to jointly register (that is, the current registration It is multi-view registration, and the X-Ray images of AP and LAT are used as input values for registration). Can this be achieved using DiffDRR? (I have tried modifying it on DiffPose. For example
image , but the effect seems not ideal), I think if we can combine AP and LAT images, we may be able to solve the first problem. I have checked many papers, but there is no relevant research. If you can give me suggestions, I will Thank you very much!