using diffpose for my data

[sarbabi]

Hi,

We have CT and Xray images of lower extremity (hip to toes). we would like to register CT to Xray for this we thought to use DiffPose. I attach here one CT and Xray example images.

my question is -can I use DiffPose for this purpose? -Do I need training or can I run DiffPose for registration? -especially I see big computing demands of DiffPose for example 6 A6000 that I do not have access to, can I use it on my local PC or a server? -As you see in the images CT and Xray show different angles between legs (Xray is weightbearing while CT is lying) so I believe for doing registration I need to do it per bone separately (femur, tibia). is it a correct approach?

Thank you in advance. Saeed

[eigenvivek]

Hi @sarbabi, you can totally use DiffPose to do this registration. Although, depending on how many X-ray/CT pairs you want to register, it might easier to simply use DiffDRR to solve your problem. DiffPose is most useful if you want to align multiple X-ray scans from the same patient to their CT scan (as you have to train a pose regression model for every patient). If you roughly know the alignment of every frontal pose (e.g., like the pose we found before), you can simply perform the registration with DiffDRR.

I see big computing demands of DiffPose for example 6 A6000

DiffPose and DiffDRR only requires a single GPU. The reason we use 6 GPUs in DiffPose is because the pelvic dataset has 6 subjects. So, in the experiments in this repository, I'm simply parallelize the training of the subject-specific models. Also, you don't need a particularly powerful GPU. I've run DiffDRR-based registration on 2080s and 3090s extensively. The A6000 was needed to train the patient-specific pose regression CNN (although I've found ways to train equivalently accurate models with less compute requirements - happy to discuss this more).

As you see in the images CT and Xray show different angles between legs (Xray is weightbearing while CT is lying) so I believe for doing registration I need to do it per bone separately (femur, tibia). is it a correct approach?

Yes, this is likely the correct approach. In DiffPose, we assume there is one rigid transform that describes the rototranslations of all anatomical objects in the scene. However, your anatomy is composed of multiple rigid objects, each of which can have their own transformation. So yes, you are correct that you would need to register each bone separately. In theory, this should be very easy to do with DiffDRR, provided you have a segmentation mask, as we can render each structure separately. I've never tried to do this myself, so there may be some edge cases that you discover while trying to implement this. If that happens, I'd be very happy to help make it work.

[eigenvivek]

Also that right femur looks gnarly. Is that a fracture or is it some other pathology?

[sarbabi]

Dear @eigenvivek , thank you so much for your extensive reply. it is great to know that I can use DiffPose or in my case since I have only one Xray per CT I can use DiffDRR, and especially that I know a rough estimation of frontal pose like the previous question. great that I can use in on my single GPU. and Thank you for confirming my approach is correct. The patient had malaligned legs as a disease and needed surgery. I just checked in DiffDRR tutorials for registration and I saw this https://vivekg.dev/DiffDRR/tutorials/registration.html. Thank you. but I was still not sure why the ground_truth is a DRR and not an Xray? I thought it should open both CT and Xray from file and try to fit moving DRR on the Xray, but it says: ground_truth = drr(gt_pose).

[eigenvivek]

but I was still not sure why the ground_truth is a DRR and not an Xray

The ground truth was a DRR just for demonstration purposes. In real world settings, the ground truth would be a real X-ray. For example, you can see how registration with DiffDRR is implemented in the DiffPose experiments here, where the ground truth is a real X-ray.

[sarbabi]

That is clear. Thanks a lot.

[eigenvivek]

You’re welcome!