Connecting Philips iU22 and NDI Aurora tracker

[TomJoosten]

Hello,

I am trying to connect a Philips iU22 ultrasound device's 2D probe and a NDI Aurora tracker system to perform a temporal and spatial calibration in fCal. To this extent, I have used the example "PlusServer: NDI Aurora tracker" which connects correctly and detects my needle.

A picture of my set-up. iU22 in the back, the white box attached to the arm is the Aurora field generator. The needle is in front of the keyboard.

Now onto the hard part, gathering the US data from the iU22. Currently, we have some module in MeVisLab that is able to gather the iU22 B-scans via a laptop and run an OpenIGTLink server on a certain IP and port. I am however struggling with getting this connection working. I think I am just framegrabbing over an OpenIGTLink.

PlusDeviceSet_Server_PhilipsUS.xml

<Device
  Id="USDevice"
  Type="USTracker"
  ToolReferenceFrame="Tracker" >
  <DataSources>
    <DataSource Type="Tool" Id="Needle" PortName="0" />
  </DataSources>
  <OutputChannels>
    <OutputChannel Id="USStream">
      <DataSource Id="Needle" />
    </OutputChannel>
  </OutputChannels>
</Device>

The problem is (as far as my knowledge extends) in the part where I define the US-device. I can't however find out how to fix this. Could someone give me some pointers? (Either on tutorials or documentation)

Kind regards,

Tom Joosten

[TomJoosten]

I have found out that removing the latter PlusOpenIGTLinkServer (which is stupid, I shouldn't create another server!) and adding the following device solves the problem:

<Device
  Id="USDevice" 
  Type="OpenIGTLinkVideo"
  ServerAddress="127.0.0.1"
  ServerPort="18945" >
  <DataSources>
    <DataSource Type="Video" Id="Video" PortUsImageOrientation="MF" />
  </DataSources>
  <OutputChannels>
    <OutputChannel Id="USStream" VideoDataSourceId="Video" />
  </OutputChannels>
</Device>

(i.e., does not give any errors) I am keeping this open as I am not finished yet, I still need to get the images from this Video and then get perform the calibration steps. How can I broadcast this with the line segmentation like is done in the time calibration example?

[adamrankin]

Hi Tom,

If I understand correctly, you're looking to do a temporal calibration? Also a probe calibration? If so, you can use fCal or 3D Slicer depending on your desired method of calibration (temporal is fCal only). fCal uses a Z-wire method, and Slicer uses a hand-eye method. If you want to use fCal, you have to create (I recommend modifying an existing) configuration file for fCal.

See https://github.com/PlusToolkit/PlusLibData/blob/master/ConfigFiles/PlusDeviceSet_fCal_Ultrasonix_L14-5_NDIAurora_2.0.xml as an example.

In that config file, you'd just have to change the imaging device (as you've already figured out) and update the parameters of the aurora device entry.

[TomJoosten]

Hi Adam, thank you for your quick reply!

I am looking for the transformation matrices of a spatial calibration as well as doing a temporal calibration, but I think that once I have the temporal calibration figured out, the spatial calibration will be self-explanatory. One of the Aurora sensors will be mounted on the ultrasound probe in order to gather position and orientation data for every 2D image and combine these into 3D images. We have the N-wire phantoms you supplied 3D-printed. I will update this with any questions whenever they arise. :-)

Kind regards,

Tom

[adamrankin]

Ok, this is our most common use-case. You can use fCal for all of your steps.

[adamrankin]

If you have a braided wire, you will get more precise fiducials than with solid wire.

[TomJoosten]

Hi there,

There are some issues on the server side of the OpenIGTLink and I can't seem to collect the ultrasound images yet, but nevertheless I have some questions on the PLUS side as well. If we speak about the temporal calibration, I need the following (please correct me if I'm wrong):

• The Aurora sensor position, so it can be tracked while I make a sine-movement. (This is already handled perfectly by the example provided in fCal)
• The ultrasound image stream, so a line of high reflection can be segmented for every frame and the distance to the reflective surface can be acquired. I found out that I need to tell fCal that what I am acquiring is an image stream and show this image as is being done in the "fCal: Simulation mode for temporal calibration" example. For now, I just copied the tag and the tag, of which I changed some parameters to match my configuration. (See attached file)

I see that for the transducermodel, some probe model (Probe_EEL14-5_38.stl) is used. However, I am using a different probe (Philips L12-5). Do I also use this model? Why do I need such a 3D model of my probe? I found some PowerPoint made by Paweena U-Thainual about the probe calibration which states that you can use the file both for C5 and L14 scanners, so what about my (different manufacturer) L12?

Furthermore, I will need to do a stylus calibration as well for this, for which I will also need to include some information in my .XML-file.

Please find my config-file attached as a .TXT-file: PlusDeviceSet_Server_PhilipsUS.xml

Kind regards,

Tom

[adamrankin]

The probe stl model is just for display purposes.

Your aurora configuration also needs a probe sensor entry

<Device
  Id="TrackerDevice" 
  Type="AuroraTracker"
  SerialPort="6"
  ToolReferenceFrame="Tracker" >
  <DataSources>
    <DataSource Type="Tool" Id="Needle" PortName="0" />
    **<DataSource Type="Tool" Id="Probe" PortName="1" />**
  </DataSources>
  <OutputChannels>
    <OutputChannel Id="TrackerStream">
      <DataSource Id="Needle" />
      **<DataSource Id="Probe" />**
    </OutputChannel>
  </OutputChannels>
</Device>

Your server tag will need to send images as well (Plus will time-sync images and transforms).

  <MessageTypes> 
    <Message Type="TRANSFORM" />
    <Message Type="IMAGE" />
  </MessageTypes>

There are other changes for sending images, I direct you to the config files for examples.

[TomJoosten]

A small update from my side: I have been able to send an artificially loaded image into Slicer3D over the OpenIGTLink via MeVisLab, and will try to connect the ultrasound device tomorrow! It looks like it is coming all together now. :-)

Thank you for your quick and helpful responses Adam!

[TomJoosten]

Hello there,

I managed to get the ultrasound images into fCal, see the screenshot below of the link between MeVisLab and 3D Slicer:

With the knowledge that the connection worked fine with Slicer, I tried getting the images into fCal. Therefore, I added a device:

<Device 
  Id="TrackedVideoDevice" 
  Type="VirtualMixer" >
  <InputChannels>
    <InputChannel Id="TrackerStream" />
    <InputChannel Id="VideoStream" />
  </InputChannels>
  <OutputChannels>
    <OutputChannel Id="TrackedVideoStream"/>
  </OutputChannels>
</Device>

Which seems to work. However, I am not sure why this works and displays my video properly:

Could you please explain this to me? How does combining the TrackerStream (Which includes the Probe, Stylus and Reference) and the VideoStream (Which includes the US image) fix the problem of anything displaying? What am I actually doing in this step?

As a reference to others, my XML-file so far: PlusDeviceSet_Server_PhilipsUS.xml

Kind regards,

Tom

[adamrankin]

Most importantly, the device you added has <OutputChannel Id="TrackedVideoStream"/> which is the input to other elements in that file.

In fCal, you can change the channel that is displayed, and the default is selected in the tag. The default in your file was TrackedVideoStream which didn't exist until you added that device back in.

[TomJoosten]

Hi there, I'm still trying to get my set-up to work.

After getting the stylus calibration (of which I found that using the smaller holes in my printed version of the fCal 1.2 model gets me to a lower error as it keeps my stylus tip in a steadier position than using the somewhat larger holes in my printed version of the fCal 3.1 model) and temporal calibration between iU22 and NDI Aurora (of which the standard deviation was really low :-) ) I am now attempting to do a phantom registration.

To this extent, I have measured my fCal 3.1 phantom and applied the coordinate system supplied by your wiki pages. I.E. x is increasing from A to Q, y is increasing from front to back and z is decreasing from 6 to 14 where the origin is on the back of the front plate where the A5 wire comes out. Coordinates of a point are then given as (x,y,z).

Here is my phantom definition: ` <Description Name="fCAL" Type="Multi-N" Version="3.1" WiringVersion="1.0" Institution="Queen's University PerkLab" />

`

However, when I want to do a phantom registration with landmark detection and click the "Start Landmark Detection" button, the "Record point" button does not activate, as depicted in the screenshot below:

I also added this part to my config-file such that the correct transforms are computed:

<vtkPlusPhantomLandmarkRegistrationAlgo PhantomCoordinateFrame="Phantom" ReferenceCoordinateFrame="Reference" StylusTipCoordinateFrame="StylusTip" />

So my questions are:

• What could be the possible reason that the "Record point" button does not activate?
• Are my measurements of the phantom somewhat correct? Can I check this by opening the 3D-model in some 3D CAD software and checking all the landmarks or are there any documented landmark positions? I can find them of the fCal 2.0 phantom, but am yet to find the coordinates of the fCal 3.1 phantom.

Kind regards,

Tom

[adamrankin]

I am guessing that the phantom definition requires the wire configurations. Check your error log (the red circle in the corner) for more information.

I am not super familiar with the fCal 3.1, @lassoan any thoughts?

[lassoan]

fCal 3 phantom is for calibration large-field-of-view probes. What probe do you use? How wide it is / what is the angle range? What is the imaging depth?

[TomJoosten]

I am using https://www.usa.philips.com/healthcare/product/HC989605418551/l12-5-50mm for the acquisition of the US images, we wish to image the spinous processes in the L1-L5 region of the lower lumbar spine. (So that's not that deep) I am currently just trying to get everything working and grasping the main concepts with stuff we have laying around right now, and therefore using sub-optimal equipment.

I am however, after checking the error log (thank you Adam), able to visualize what is happening and then (apparently it is really necessary to have a visual representation as well) perform the landmark detection.

|ERROR|331.906000| Unable to retrieve phantom model by ID. Is the phantom model ID well defined?| in E:\D\PSNP64b\PlusApp\fCal\Toolboxes\QPhantomRegistrationToolbox.cxx(213) |ERROR|331.909000| Initializing phantom registration visualization failed!| in E:\D\PSNP64b\PlusApp\fCal\Toolboxes\QPhantomRegistrationToolbox.cxx(568) |INFO|331.915000| PhantomToReference transform is absent, registration needs to be performed| in E:\D\PSNP64b\PlusApp\fCal\Toolboxes\QPhantomRegistrationToolbox.cxx(398) |INFO|331.921000| PhantomToReference transform is absent, registration needs to be performed| in E:\D\PSNP64b\PlusApp\fCal\Toolboxes\QPhantomRegistrationToolbox.cxx(398) |INFO|331.924000| Toolbox changed to Phantom registration| in E:\D\PSNP64b\PlusApp\fCal\fCalMainWindow.cxx(330) |ERROR|354.901000| No stylus tip displayable objects could be found!| in E:\D\PSNP64b\PlusApp\fCal\Toolboxes\QPhantomRegistrationToolbox.cxx(1062)

I acted as if I have a fCal 2.0 phantom and used an example supplied in fCal. I'll print that in the weekend I suppose. If I were to use the fCal 3.1 phantom, the ModelToObjectTransform has to be given. I think this is not the same as the fCal 2.0, which is given by:

ModelToObjectTransform=" 1 0 0 -35.0 0 1 0 -10.0 0 0 1 -5.0 0 0 0 1"

Is there any documentation on these parameters?

Kind regards,

Tom

[adamrankin]

I don't think so. Creating your own wiring/phantom configuration is considered an expert skill. That transform is the mapping of the origin in phantom space to the origin in model space.

[TomJoosten]

Hi there,

I've gotten all the components of the calibration to work. Thank you for your help Adam! Some notes on things I ran into, if others experience any trouble:

• The <Segmentation /> tag should be present when performing calibration and tuning the segmentation parameters. Furthermore, all the parameters should be present in the <Segmentation /> tag, otherwise the software will not be able to "Apply and close" or "Save and close".
• Apparently tables with a small metal beam can already distort the EM tracker in such a way that errors blow up.
• Check the pixel size of the incoming ultrasound image. Mine was off a bit due to selecting a different depth setting on the ultrasound device. It should be correct in the <Segmentation /> tag as this is used for the segmentation.

Now onto the acquisition of the data! Thanks again!

Kind regards,

Tom

[adamrankin]

I'm glad you were able to get it up and running!

[lassoan]

@Sunderlandkyl Could you follow up on these noted:

The tag should be present when performing calibration and tuning the segmentation parameters.

Check if we log an error message saying that calibration needs this xml element. Maybe do segmentation with default settings if this element is missing.

Apparently tables with a small metal beam can already distort the EM tracker in such a way that errors blow up

Apparently, this may not be obvious for everybody, so make sure it is explicitly mentioned on the "why my calibration error is too big" section in the Plus user manual.

Check the pixel size of the incoming ultrasound image. Mine was off a bit due to selecting a different depth setting on the ultrasound device. It should be correct in the tag as this is used for the segmentation.

I don't know how to make this more clear (approximate spacing is determined by placing the two measurement lines over known distances).

[Sunderlandkyl]

@lassoan Sure, I'll follow up.

[Sunderlandkyl]

@lassoan There is already an error, or at least warning logged when a Segmentation element is not included (in "vtkPlusImageVisualizer.cxx", "QSpatialCalibrationToolbox.cxx", and "QPlusSegmentationParameterDialog.cxx").

Should the default values be the ones from the sample calibration files?: <Segmentation ApproximateSpacingMmPerPixel="0.078" MorphologicalOpeningCircleRadiusMm="0.27" MorphologicalOpeningBarSizeMm="2.0" ClipRectangleOrigin="27 27" ClipRectangleSize="766 562" MaxLinePairDistanceErrorPercent="10" AngleToleranceDegrees="10" MaxAngleDifferenceDegrees="10" MinThetaDegrees="-70" MaxThetaDegrees="70" MaxLineShiftMm="10.0" ThresholdImagePercent="10" CollinearPointsMaxDistanceFromLineMm="0.6" UseOriginalImageIntensityForDotIntensityScore="FALSE" NumberOfMaximumFiducialPointCandidates="20" />

[lassoan]

Should the default values be the ones from the sample calibration files?

Yes. Thank you!