Use of 2D GT landmarks

[pimatysiak]

Hi, thank you for your paper and code, very nice work!

I was wondering if you could illuminate me as to where in your code you access (or use) the ground truth 2D landmarks for the projections when performing the registration test.

I am asking this to test your code with my own annotated CT and generated DRR with manual landmark annotations. I notice I can run the code and have some results without ever specifying where those ground truth 2D landmarks should be collected from, which is odd.

Thank you in advance for this.

[eigenvivek]

Hi @pimatysiak, the two datasets we evaluate in this work don't have ground truth 2D landmarks per se. Instead, the authors of these datasets annotated landmarks in each 3D CT and projected these into 2D using the ground truth extrinsic/intrinsic matrices. In the DeepFluoro dataset, ground truth poses were determined with an offline global registration processes. In the Ljubljana dataset, ground truth poses were determined by registering implanted fiducials - unfortunately these 2D/3D landmarks are not available in the dataset.

This was something that confused me too when I started working with these datasets. This issue may help clarify: https://github.com/rg2/DeepFluoroLabeling-IPCAI2020/issues/4

[pimatysiak]

Hi Vivek, thanks for the quick reply!

Specifically for the DeepFluoro dataset I know the landmarks were computed automatically, but the paper specifies that they were manually inspected and verified, which is why I considered them to be "ground truth", and they are included in the dataset.

I guess if you do not use them for the validation and instead project landmarks on the provided fluoro images from the 3D landmarks, it's fine, I guess that was just a misunderstanding on my part.

Would you think it possible to still plug in those 2D ground truth landmarks into the registration framework, so I can rule out bad automated landmarks if the results I get are unsatisfactory? Could you point out which section of the code I should modify for this? I notice you have a _get_2dfiducials method but I don't see it used.. unless I'm blind.

[eigenvivek]

get_2d_fiducials is used for visualization purposes - it's been called whenever you see landmarks in figures in the paper.

The 2D landmarks aren't only used for figures, however. They're also used in the validation function that computes registration error (see, the Evaluator class). I could have unified their implementation to use the same helper function ... something to simplify in future implementation!

[pimatysiak]

Ah I see, that's why I didn't find it in the validation. Thanks!

Forgive me for being annoying, but could you specify where you load the 2D landmarks? I noticed in your Evaluator class that you indeed load what you call fiducials from the specimen, but it looks to me that the ones that are loaded are only the 3D landmarks from the patient, not the individual 2D landmarks from each fluoro image used in the evaluation.. Unless I am once again mistaken?

[eigenvivek]

I don't load the 2D landmarks from the DeepFluoro dataset. Since they're just the perspective projection of the fiducials, I just compute them myself. For sanity, here's a check that the two are equal:

import torch
from diffpose.calibration import perspective_projection
from diffpose.deepfluoro import DeepFluoroDataset, convert_diffdrr_to_deepfluoro

idx = 1  # Specimen ID
jdx = 0  # X-ray ID

specimen = DeepFluoroDataset(idx)

# From DiffPose
_, pose = specimen[jdx]
pose = convert_diffdrr_to_deepfluoro(specimen, pose)
perspective_projection(pose, specimen.intrinsic, specimen.fiducials)

# tensor([[[1599.6984,  639.4305],
#          [  29.8002,  625.9924],
#          [1409.0881, 1118.6995],
#          [ 236.0820, 1138.9253],
#          [1314.6082,  673.6483],
#          [ 386.6854,  685.3052],
#          [1156.6296, 1393.9117],
#          [ 563.0671, 1408.9928],
#          [ 858.9445, 1423.1156],
#          [ 805.6584, 1422.8655],
#          [ 985.3099, 1336.1173],
#          [ 688.2995, 1325.3735],
#          [ 845.8329, 1303.7023],
#          [ 796.3127, 1305.0377]]])

# From DeepFluoro
torch.tensor([x[:] for x in  specimen.projections[f"{jdx:03d}/gt-landmarks"].values()]).squeeze()

# tensor([[1599.6986,  639.4305],
#         [  29.8006,  625.9923],
#         [1409.0884, 1118.6995],
#         [ 236.0822, 1138.9249],
#         [1314.6084,  673.6480],
#         [ 386.6857,  685.3053],
#         [1156.6298, 1393.9119],
#         [ 563.0674, 1408.9928],
#         [ 858.9448, 1423.1151],
#         [ 805.6586, 1422.8654],
#         [ 985.3102, 1336.1173],
#         [ 688.2997, 1325.3733],
#         [ 845.8331, 1303.7023],
#         [ 796.3130, 1305.0375]])

They're equal up to floating point error, which probably contributes ~10^-3 mm of registration error.

[pimatysiak]

I understand now, thank you for your patience!

[eigenvivek]

no problem at all, happy to answer any questions. hope it's helpful for your work!