search

HL2-DINO / DINO-DLL

A C++ HoloLens 2 DLL used for detecting and tracking infrared markers with the Research Mode API
https://hl2-dino.github.io/DINO-DLL/
BSD 3-Clause "New" or "Revised" License
15 stars 1 forks source link

blob frame #2

Closed jakubaugusty closed 1 month ago

jakubaugusty commented 1 month ago

Hi, Very interesting project, congratulations. Could you guide me on how to display the last frame on the hololens screen? I mean exactly this: image

hiqb217 commented 1 month ago

Hi!

Sorry, just to clarify one thing, are you asking how you could recreate that particular image (i.e. labelled blobs?)

Or is your question more about data transfer to the HoloLens, e.g. you want to know how to send that image from the DLL to Unity so you can see it when wearing the HoloLens?

jakubaugusty commented 1 month ago

Thanks for the quick response!

Yes, that's the second issue. I tested your ready-made solution for Unity 2021 and everything seems to work fine, however, in this version on HoloLens 2 only 2 frames are visible: depth frame and ab frame. I would like to be able to see the blob detection frame like in the screenshot attached above.

hiqb217 commented 1 month ago

So the .GIF in the README is one I'd made as a one-off for demo purposes. But you should be able to do something similar yourself with a few modifications to the code.

What you see in the screenshot is basically the AB frame, but what I'd done (from my rough memory) was to call some OpenCV functions to annotate the image: 1) cv::threshold to binarize the image so you only keep the bright part of the image 2) cv::circle and draw a circle around each detected blob

At present, two 8-bit images are passed to Unity for display purposes, a processed version of the depth image and AB image (where there are cross markers drawn on the centre of blobs associated with tools).

If you wanted to draw circles/crosses on every detected blob (e.g. random blobs which aren't attached to a tool), it should be fairly easy to do just by adding some code yourself.

In this part of the code, you iterate over any detected tools, and then draw crosses over the centres of any blobs associated with the tool: https://github.com/HL2-DINO/DINO-DLL/blob/2c27685b3af3f213d2a657750490242893b41b22/HL2DinoPlugin/src/IRImageProcUtils.cpp#L272-L282

The variable m_cache_frameBlobInfo, which is used in the file Holo2IRTracker.cpp is updated each frame so it stashes a list/vector of all detected blobs, and it contains 2D pixel coordinates of all detected blob centres.

You could adapt the function LabelImageWithToolDictData above, and change it so it also accepts m_cache_frameBlobInfo as one of its arguments, and then iterate over this and similarly draw circles/crosses. The result of this will be that the AB frame will now have extra annotations.

Remember to recompile the project and copy over your new .dll and .winmd file to Unity, and if you rebuild everything it should show any changes you make.

jakubaugusty commented 1 month ago

Thank you very much for the explanation!