Given extrinsic matrix and intrinsic matrix

[Suru4t]

Hi thanks for your fascinating work at first. I'm trying to get the projection when I already have both extrinsic matrix and intrinsic matrix for projection. As usually defined, my extrinsic matrix is the rigid mapping from world frame to camera projective frame(4*4 matrix), and intrinsic matrix is projective mapping from camera projective frame onto image plane (3x3 matrix). After trying in many ways, I only get empty fluoroscopy result which is totally black. What is the correct way to use these projection paras in Deepdrr?

[benjamindkilleen]

It's hard to know what the problem is without a minimal working example. Can you provide one, where the volume is manually instantiated, e.g. with numpy?

[Suru4t]

Info of the volume I use:

my code is like this:

   patient = deepdrr.Volume.from_nifti("volumeCT.nii.gz",
        use_thresholding=True
    )
    patient.faceup()
    # define the simulated C-arm
    carm = deepdrr.MobileCArm(
        world_from_device=geo.frame_transform(extrinsic),
        sensor_height=1536, sensor_width=1536, pixel_size=0.194)

    with Projector(patient,
                   carm=carm,
                   camera_intrinsics=geo.CameraIntrinsicTransform(intrinsic)
                   ) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to `{path}")

[Suru4t]

Info of the volume I use:

my code is like this:

   patient = deepdrr.Volume.from_nifti("volumeCT.nii.gz",
        use_thresholding=True
    )
    patient.faceup()
    # define the simulated C-arm
    carm = deepdrr.MobileCArm(
        world_from_device=geo.frame_transform(extrinsic),
        sensor_height=1536, sensor_width=1536, pixel_size=0.194)

    with Projector(patient,
                   carm=carm,
                   camera_intrinsics=geo.CameraIntrinsicTransform(intrinsic)
                   ) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to `{path}")

extrinsic is a 4x4 np array and intrinsic is 3x3 np array

[benjamindkilleen]

Since you already know your camera matrix, you don't need to create a Device (such as MobileCArm). These are convenience classes for simulating real C-arms and deriving the corresponding projections. The world_from_device transform is not the extrinsic matrix but actually the transform to the base of the C-arm.

You can provide the projection matrix directly:

from deepdrr import geo
camera_intrinsics = geo.CameraIntrinsicTransform(intrinsic, sensor_height=1536, sensor_width=1536)
projector = Projector(patient, camera_intrinsics=camera_intrinsics)
with projector:
    index_from_world = geo.CameraProjection(extrinsic, intrinsic)
    image = projector(index_from_world)
    ...

In future, a minimal working example is a piece of code I can actually run without editing, so it would include imports and volume definitions. That way, I can actually run your code to see if it works. The above is just a guess at the main problem, but it's hard to say without a minimum working example.

[Suru4t]

Thank you for your patient explanation, Benjamin. Now I roughly understand how to use the extrinsic matrix and the intrinsic matrix for DeepDRR projection. However, I found that the image projected in this way is still completely black. Maybe the cause is the incorrect way of my directly using the extrinsic matrix that the dataset provided. I put the code below.

To be more specific, I want to use the DeepFluoro dataset for projection. In this dataset, it is mentioned that the coordinate frame is defined as follows：https://github.com/rg2/DeepFluoroLabeling-IPCAI2020/blob/master/hdf5_layouts/Readme.md#notes-on-spatial-transforms In this page, DeepFluoro datasets mentioned that the projective/camera coordinate frame origin is located at the X-ray source. Is it possible that this is the cause?

(code example)

import os
import deepdrr
from deepdrr import geo
from deepdrr.utils import test_utils, image_utils
from deepdrr.projector import Projector
import numpy as np

output_dir = 'output'
extrinsic = np.array([[0.0014173903, 5.773302E-6, 0.9999991, -157.58221],
                          [-0.99999076, -0.004075836, 0.0014174007, -2.6757495],
                          [0.0040758415, -0.99999183, -0.0, -631.7765],
                          [0.0, 0.0, 0.0, 1.0]
                          ])
    intrinsic = np.array([[-5257.732, 0.0, 767.5],
                          [0.0, -5257.732, 767.5],
                          [0.0, 0.0, 1.0]])
    patient = deepdrr.Volume.from_nifti(
        r"data/ipcai_2020_full_res_data/17-1882/volumeCT.nii.gz",
        use_thresholding=True,
    )
    camera_intrinsics = geo.CameraIntrinsicTransform(intrinsic, sensor_height=1536, sensor_width=1536)
    # project in the AP view
    with Projector(patient,
                   camera_intrinsics=camera_intrinsics
                   ) as projector:
        index_from_world = geo.CameraProjection(intrinsic, extrinsic)
        image = projector(index_from_world)
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to {path}")

[benjamindkilleen]

The problem most likely has to do with how you are loading the DeepFluoro dataset, where the "extrinsic" may not refer to the same matrix as DeepDRR (Z coordinate might be flipped).

Can you create a working example with a "test" volume? I'm not able to run the above without downloading the DeepFluoro dataset, so it's hard to say what's wrong.

[Suru4t]

I find that my code's output DRR is not all black(value of which is 0), there is one pixel valued 1 in the center of the DRR picture.

[benjamindkilleen]

Can you try this:

import os
import deepdrr
from deepdrr import geo
from deepdrr.utils import test_utils, image_utils
from deepdrr.projector import Projector
import numpy as np

output_dir = 'output'
patient = deepdrr.Volume.from_nifti(
        r"data/ipcai_2020_full_res_data/17-1882/volumeCT.nii.gz",
        use_thresholding=True,
)
patient.supine()
device=deepdrr.SimpleDevice()
superior_in_world = patient.world_from_anatomical @ geo.v(0, 0, 1)
anterior_in_world = patient.world_from_anatomical @ geo.v(0, 1, 0)
device.align(point=patient.center_in_world, direction=anterior_in_world, up=superior_in_world)
# project in the AP view
with Projector(patient, device=device, neglog=True, intensity_upper_bound=10) as projector:
        image = projector()
        path = output_dir + "/example_projector.png"
        image_utils.save(path, image)
        print(f"saved example projection image to {path}")

If that succeeds, then the issue isn't with DeepDRR but rather with your interpretation of the intrinsic/extrinsic from Rob's data.

[Suru4t]

Thanks a lot. I think its my incorrect interpretation of the intrinsic/extrinsic that I use, I will try other ways.