Beginner : tracked 3D ultrasound with Optitrack system and CAM LINK camera

[AurelieSar]

Hi, I am sorry in advance if my questions are beginner questions... Since a few days, I am desperately trying to use Slicer 3D and Plus Toolkit to do 3D ultrasound using an Optitrack motion capture system, a rigid body of 4 markers attached to my US (SuperShearing Imaging) probe, and a video capture USB device (CAM LINK) collecting the video feed from the US. I first tried to write my Optitrack config file and linking it to Slicer 3D. All seemed to work. Now I am trying to visualise the video feed from the CAM LINK in Slicer 3D. I don't know which config file to use : I tried both config files "video for Windows video capture device" and "media foundation video capture device". The connections are successful but then in Slicer, after adding and activating the IGTLConnector, and changing the parameters in Plus Server Control, I cannot seem to visualise the video feed ... How can I check if my video feed is working ? Thanks in advance, Aurélie

[AurelieSar]

The log file is indicating :

081720_165226.125|WARNING|118.076000|SERVER> No data is broadcasted, as no data is available yet.| in :\D\PSNP64b\PlusLib\src\PlusServer\vtkPlusOpenIGTLinkServer.cxx(417)

I previously checked that in the software OBS I was correctly seeing my video feed from the US through the CAM LINK.

[adamrankin]

Could you upload your config file that includes the video device?

[AurelieSar]

I actually tried to use the two default configuration files of Plus: PlusDeviceSet_Server_VfwVideoCapture.xml & PlusDeviceSet_Server_MmfVideoCapture.xml before changing any data. I checked in OBS Studio and the video format of my USB capture device is YUY2 as indicated in the config files. I am not sure which one to use. My video capture device is the following : https://www.elgato.com/en/gaming/cam-link-4k default_config_files.zip

[adamrankin]

What source are you using in OBS studio?

[AurelieSar]

I select the source "video capture device" and then the name of the device is "Cam Link 4K"

[adamrankin]

Ok, media foundation is the right path to go down. Could you scan the log output for any error messages or any other warning messages?

Or, could upload the whole log file?

[Sunderlandkyl]

This might be related: https://discourse.slicer.org/t/optitrack-duo-and-external-webcam-problem/12730/7

HI Angel, Glad to help! This is a known issue that can’t be addressed, unfortunately. You’ll be unable to run both the web camera and the Duo on the same computer. Best, Ryan OptiTrack Support

[AurelieSar]

Thank you all for your quick responses ! @Sunderlandkyl I do not have the Duo Optitrack but Flex 13 Optitrack optical cameras so I should be good ;-) @adamrankin with your information, I used the config file in the folder attached. I changed the resolution at 1280x720 because I saw in OBS that it was the resolution of my capture device. Then I launched the Plus Server. All is good till then, I have the log file 1 attached with no errors or warnings. The connection works only with the device ID "0", not with "1" or "2". I have no other cameras on this computer (no webcam). But then when I launch Slicer, I go to "OpenIGTLinkIF", I add the connector and activate it, the status goes to "ON", and I have the log file 2 attached, with warnings. The main warning is : "no data is broadcasted, as no data is available yet." files.zip Thanks ! Aurélie

[adamrankin]

Device 0 is to be expected if you only have 1 camera (of any type) plugged in.

Just in case, although it should work anyways. Could you try with the PlusDeviceSet_Server_MmfColorVideoCapture.xml config file? (adjusting the resolution parameters to match your camera)

Edit: it should be working, this is unusual.

[AurelieSar]

With the MmfColorVideoCapture.xml, when I tried the default resolution, the log file confirmed me that it was receiving CAM LINK 4K on the Device 0 and that I needed to change the resolution. After changing the resolution to 1280*720 it worked. I feel like maybe all is working well on the Plus Server side but I am doing (or not) something wrong in 3D Slicer. I am not sure how to display the video feed. Here attached two photos from 3D Slider, one from the config file, one from the log (which displays the same warning as before), and one from the Plus Server. Thanks ! Aurélie

[adamrankin]

Edit: no nevermind

There is no image entry under "IN" in Slicer. This means that Plus isn't sending images for some reason.

[adamrankin]

Is OBS closed when you try and run Plus/Slicer?

[adamrankin]

And could you check for any zombie PlusServer.exe processes

[AurelieSar]

Yes, OBS is closed when I try and run Plus & Slicer No, there is no zombie PlusServer.exe process I was indeed surprised there was no image entry under "IN", but it is normal ?

[AurelieSar]

I realised that I could not record with OBS due to old NVIDIA drivers. I updated them and rebooted the computer. I now can record with OBS, however the situation is the same for Plus & 3D Slicer.

[adamrankin]

I can confirm that MMF - color is working with my webcam. I honestly don't know what's happening. The next step is to build in debug and work your way back to where it's failing.

[AurelieSar]

Actually, I must be doing something wrong, since I just tried with another computer equipped with a webcam and I cannot also display the video feed, the warning message in the log file is exactly the same ! Can you upload a capture of your Slicer interface or describe to me what you are doing after opening Slicer ? I must be missing something.

Just for information:

• I am using Slicer 4.11.0 since I could not see the "OpenIGTLinkIF" module in Slicer 4.10.2, even after installation of the extensions.
• With a "motion capture only" configuration file, all is working fine, I succeeded in doing a pivot calibration. The only thing missing is the video from the ultrasound...

Regarding your message I am not competent enough to build in debug mode ... :/

[lassoan]

Maybe your image capture device can only provide compressed video stream. You can try to capture the image using a virtual camera as described in the documentation (http://perk-software.cs.queensu.ca/plus/doc/nightly/user/DeviceMicrosoftMediaFoundation.html).

[AurelieSar]

Thanks everyone for your help. No I checked, and the CAM LINK uses the UVC protocol so it should work, as it is working in OBS like my webcam. And I do not succeed in seeing my video feed from my webcam either in Slicer, so I must be forgetting a step in the setup ... For now, I launch the Plus Server, then launch Slicer, go the "OpenIGTLinkIF", add the connector and activate it, its status turns ON, am I missing something ? If after that I launch OBS, then I cannot access to the video feed of either the webcam or the CAM LINK anymore, so the feed must be successfully sent to Slicer ...

[adamrankin]

I think the last thing we could try is a zoom session with shared desktop and I could try and investigate. At this point I don't know why this is happening, sorry. :(

[AurelieSar]

So ... I tried to upload my webcam feed into Slicer at home and it worked just fine .. making me believe that the University context was the problem so I tried and closed the antivirus we have at Univ and ... it worked !!! The antivirus was the culprit, blocking the video feed (but surprisingly not the Optitrack feed). I still need to correctly setup the image but I am very glad this is working (see picture). Thanks a lot for your reactivity !

Next step is to try the temporal and spatial calibration (with the stylus). Just before you close this topic, can you tell me if this is possible to do only the temporal calibration with fCal and then to switch to Slicer for the pivot and spatial calibrations ? I did not find a tutorial about the temporal calibration using Slicer .. Thanks a lot to the team ! Aurélie

[adamrankin]

Yes, there is no temporal calib in Slicer. Just do it in fCal, save the result to a new config file, or update your previous config file with the offset, and continue in Slicer.