search

EnoxSoftware / HoloLensWithOpenCVForUnityExample

HoloLens With OpenCVforUnity Example (Support for Hololens1 and Hololens2)
MIT License
262 stars 50 forks source link

Can this example work on hololens2 (arm) ? #29

Closed EnoxSoftware closed 1 year ago

EnoxSoftware commented 3 years ago

Could someone try out if this native plugin can work with Hololens2 and replace HololensCameraStream? https://github.com/carmines/UnityPluginCollection/tree/master/CameraCapture

I haven't been able to check it out as I don't have Hololens2 at hand, but I have confirmed that I can get video preview and camera matrices successfully on Hololens.

cookieofcode commented 3 years ago

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

EnoxSoftware commented 3 years ago

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information. Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

ssyplot commented 3 years ago

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information. Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

HoloLens 2 Research Mode enables access to the raw streams on device (depth camera, gray-scale cameras, IMU). https://github.com/microsoft/HoloLens2ForCV

EnoxSoftware commented 3 years ago

Hi There

Thank you for the helpful information. I found a very interesting repository. https://github.com/qian256/HoloLensARToolKit This repository seems to include how to get the video preview and camera matrices of the camera on the Hololens2 device I was looking for.

cookieofcode commented 3 years ago

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information. Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

HoloLens 2 Research Mode enables access to the raw streams on device (depth camera, gray-scale cameras, IMU). https://github.com/microsoft/HoloLens2ForCV

The Research Mode is interesting, but it needs to be specifically enabled and currently needs Windows Insider Preview.

Hi There

Thank you for the helpful information. I found a very interesting repository. https://github.com/qian256/HoloLensARToolKit This repository seems to include how to get the video preview and camera matrices of the camera on the Hololens2 device I was looking for.

I've also discovered the HoloLensARToolKit repository, but it seems rather bloated for my purpose, as I don't need Trackers.

Aquiring and displaying a video stream with HoloLensCameraStream using MediaCapture works fine on the HoloLens 2. I'm not sure if all transformations work yet.

EnoxSoftware commented 3 years ago

The best scenario for me is for the HololensCameraStream developers to incorporate code that supports Hololens2. From looking at the latest HololensARToolKit code, it looks like there are tips to properly support Hololens2.

EnoxSoftware commented 3 years ago

This is a great post! https://github.com/VulcanTechnologies/HoloLensCameraStream/issues/63#issuecomment-718216694

nous- commented 3 years ago

@EnoxSoftware Note on that post above, I neglected to mention to transpose the output of the HololensCameraStream TryGetWorldCoords to get a column major matrix which was required for all the other steps. Updated the original post to show this happening in the plugin

EnoxSoftware commented 3 years ago

This repository shows that the CameraStream plugin works successfully on hololens2. https://github.com/abist-co-ltd/hololens-opencv-laserpointer Basically it seems that you just need to change the build architecture to ARM64.

Anyway, I don't have hololens2 at hand, so I'll put this issue on hold for a while.

cookieofcode commented 3 years ago

@EnoxSoftware The example works on HoloLens 2 with some modifications. The problem is the changed HoloLensCameraStream library and the newly introduced dependency on the CameraIntrinsics:

EnoxSoftware commented 3 years ago

@cookieofcode Thank you for the helpful information.

In the test I did on Hololens1, It was found that the combining static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview and mediaCapture.AddVideoEffectAsync () slows down significantly. Therefore, I changed to camnewnham/HoloLensCameraStream, but it was unexpected that side effects appeared on Hololens2.

aioniosfoititis commented 3 years ago

I can confirm that the changes that @cookieofcode proposes do work for holoLens 2 for most of the examples of this guide.

Although I noticed two more problems.The first one is that in the face recognition examples only in the left-Lens of the glasses the rectangles appear, and the second one is that HoloLens PhotoCapture example seams not to be working.

The total examples work fine in the unity editor.

Abdul-Mukit commented 3 years ago

None of the examples are working. I tried to follow all the instructions from the readme. As soon as I run even the License scene I see an error message on hololens in red. It says: "UnityException: destroy can only be called from main thread." For other scenes, I am getting ArgumentOutOfRangeException for the parameter named index. I am running using ARM64. I have turned on webcam capability too. In unity editors everything is fine. I had one concern. When I added the CamStream from https://github.com/camnewnham/HoloLensCameraStream I was prompted by unity that it was using obsolete API and unity would automatically update it. I don't know if that is the reason. Also I am using MRTK 2.4

cookieofcode commented 3 years ago

None of the examples are working. I tried to follow all the instructions from the readme. As soon as I run even the License scene I see an error message on hololens in red. It says: "UnityException: destroy can only be called from main thread." For other scenes, I am getting ArgumentOutOfRangeException for the parameter named index. I am running using ARM64. I have turned on webcam capability too. In unity editors everything is fine. I had one concern. When I added the CamStream from https://github.com/camnewnham/HoloLensCameraStream I was prompted by unity that it was using obsolete API and unity would automatically update it. I don't know if that is the reason. Also I am using MRTK 2.4

One problem could be the API Update, as HoloLensCameraStream contains classes named the same as in the Unity API (e.g. https://docs.unity3d.com/ScriptReference/Windows.WebCam.VideoCapture.html). Cancel the API Updater dialog and manually fix the errors.

Abdul-Mukit commented 3 years ago

Hi, @cookieofcode. Thank you for your answer. Here are a bit more details. These are the updates that auto-updated was talking about: For MatrixUsageApp.cs, ProjectionExample.cs and VideoPanelApp.cs; error CS0246: The type or namespace name 'GestureRecognizer' could not be found (are you missing a using directive or an assembly reference?) and for ProjectionExample.cs; error CS0246: The type or namespace name 'GestureRecognizer' could not be found (are you missing a using directive or an assembly reference?) I don't think they have anything to do with the Hololens error due to not calling destroy from the main thread I was originally getting. So, I imported the CamStream folder without the examples folder and I was able to avoid this API update issue.

After that, it appeared again when importing OpenCVForUnityExamples.unitypackage. These were the problems.

  1. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(12,54): error CS0234: The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA' (are you missing an assembly reference?)
  2. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(11,26): error CS0234: The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA' (are you missing an assembly reference?)
  3. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(99,42): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)

  4. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(105,31): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)

  5. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(135,30): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
  6. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(135,70): error CS0246: The type or namespace name 'PhotoCaptureFrame' could not be found (are you missing a using directive or an assembly reference?)
  7. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(200,30): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
  8. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(25,9): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)

I triggered auto-update and the errors went away. But the original error of "UnityException: destroy can only be called from the main thread." is still there in hololens and none of the examples work. Do you still think manually fixing might be the answer?

Abdul-Mukit commented 3 years ago

Only the "Check Intrinsic" scene worked, sort of. I got the following output. I am not sure if the values are correct for 896x504 case. Following is a snapshot: image

As you can see for the 896x504 case cx,cy are: 1123, 614. And the imageWidth, imageHeight from IntrinsicsMatrix is 2272x1278. . These values are not making sense to me. Would really appreciate if you could comment on this too. Thank you.

cookieofcode commented 3 years ago

The type or namespace name 'GestureRecognizer' could not be found

Looks like a missing import of the "UnityEngine.XR.WSA.Input" (instead of UnityEngine.VR.WSA.Input). Not sure about this one.

The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA'

Unity moved UnityEngine.XR.WSA.WebCam to UnityEngine.Windows.WebCam. I recommend to do the updates manually, as the API Updater may interfere with HoloLensCamStream.

As you can see for the 896x504 case cx,cy are: 1123, 614. And the imageWidth, imageHeight from IntrinsicsMatrix is 2272x1278. . These values are not making sense to me. Would really appreciate if you could comment on this too. Thank you.

I'm not familiar with this scene, but if something with the intrinisic is wrong, check my comment with the code suggestions above (https://github.com/EnoxSoftware/HoloLensWithOpenCVForUnityExample/issues/29#issuecomment-731091579).

Abdul-Mukit commented 3 years ago

Thanks again. I manually fixed the API update issues. Just needed to change to UnityEngine.Windows.WebCam in HoloLensPhotoCaptureExample.cs. However, it didn't solve the issue. Still getting the original unityException.

A little progress. Along the lines of your suggestion of using the MediaStreamType.VideoPreview, I edited the CameraIntrinsicsChecker class to use it instead of VideoRecord type. And now I get the correct intrinsic. Here is a snapshot: image

I'll try out the same in the PluginProject and see if works. :)

Abdul-Mukit commented 3 years ago

I rebuilt the plugin as suggested in https://github.com/EnoxSoftware/HoloLensWithOpenCVForUnityExample/issues/29#issuecomment-731155577 Finally, all examples were working. Thank you, @cookieofcode. So the summary is that by changing the plugin and the CameraIntrinsicsCheckerHelper.cs (for the intrinsic scene) got the project working on HL2 for me. One difference is that it only worked on the ARM64, never worked on ARM. As was mentioned by @aioniosfoititis the display is only on the left eye. Didn't check the calibration using Aruco marker scene. Overall, in all scenes, I got around 10-30fps. Works for me. Thank you. :)

A00107408 commented 3 years ago

Thank you @Abdul-Mukit @cookieofcode and @EnoxSoftware. Following your comments I got it working on HL2, with some exceptions:

In Summary:

I made the change to camnewnham/HoloLensCameraStream suggested in #29 (comment) and built the dlls.

Importing HoloLensWithOpenCVForUnityExample.unitypackage in Unity 2018.4.28f1 didn't trigger the API updater, so happy days, no errors to fix manually.

Built for ARM.

Problems:

ArUco scene works fine in both eyes, until I turn on camera preview. It flickers once or twice, then stops working.

Comic filter scene works in left eye only?

Hope this helps. Thanks again.

I can confirm that this works in Unity 2019.4.15f with MRTK 2.5.4.

Abdul-Mukit commented 3 years ago

Still getting webcam preview only on the left eye.

aioniosfoititis commented 3 years ago

@Abdul-Mukit hi, go into the mixed reality settings and go for single pass instead of single pass instanced. That fixed the issue for me

Abdul-Mukit commented 3 years ago

Thank you. That solved the problem.

Rutherfordum commented 3 years ago

Tested on HoloLens x86 and HoloLens2 ARM64 Master/Release

1) change the following line in camnewnham / HoloLensCameraStream VideoCapture.cs: static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoRecord; to static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview; and collect the solution. Replace the resulting HoloLensCameraStream.dll in the Unity Assets \ CamStream \ Plugins \ WSA project

2) change the line, in the Unity project CameraIntrinsicsCheckerHelper.cs, static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoRecord; to static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview

Thank you very much @cookieofcode and @Abdul-Mukit

EnoxSoftware commented 2 years ago

I found a very interesting Hololens2 and OpenCVForUnity integration example project. It contains a concise and simple code for using the Locatable camera on Hololens2. https://github.com/cookieofcode/LabAssistVision

AlexanderRodecape commented 2 years ago

Thanks to the contributions here I was able to get this to run on a hololens 2 built for arm64. However, on the face recognition example, the red box overlays don't seem to be in the right place when looking through the hololens. On the device portal live preview the overlay looks to be in the correct place, though. Any suggestions? Thanks for creating this repo.