Comments on "Using Azure Custom Vision Object Recognition and HoloLens to identify and label objects in 3D space"

[LocalJoost]

Original article: https://localjoost.github.io/using-azure-custom-vision-object/

[32177061]

Hello ! @LocalJoost :） I'm currently doing research on implementing object detection on hololens. This project helped me a lot, but it uses HoloToolkit and I want to run it on unity2020. Can you write a tutorial on how to upgrade the project to unity2020? Thank you very much!

[LocalJoost]

Hi, A lot has changed, and I know some students in Switzerland have managed to do this. I would have to research this as well. It's on my todo-list, but so are another 5443 things ;)

[matthew-oleary02]

Hello, I'm not sure if you still respond to comments but I am currently trying to get custom vision working on the HoloLens2 so that in my application it can identify a 3D Printer. I tried following this tutorial to get the custom vision to work but I am having a tough time trying to test it. The issue I'm having is with the step under the section titled "Take and upload images" which states: "Once you have enough images, select the Train button to start the model-training process in the cloud. This will upload all images and then start the training. The process can take up to a minute or more. A message inside the menu indicates the current progress. Once it indicates the process is complete, you can stop the application." Unfortunately, once I take the 6 pictures and press the train button I get a message that states "Please wait, uploading images" but nothing ends up happening after waiting a while. Eventually after waiting for so long, my application crashes. Would you know how to fix this and get it working? I think my issue has to do with the credentials used on Unity3D in the script called ObjectDetectionManager. For the initial part of this section, which credentials are needed to get this feature to work. The six boxes for credentials on the ObjectDetectionManager are as follows: "Azure Resource Subscription Id", "Azure Resource Group Name", "Cognitive Service Resource Name", "Resource Base Endpoint", Api Key" and "Project Id". For each credential I listed, which resource would I be taking from: the custom vision project, the custom vision resource, or the custom vision prediction resource?

[LocalJoost]

I still respond to comments, but this project is a 5 year old machine learning oriented project - AI and ML are moving SO fast this is almost archeology. This was almost from the beginning of Custom Vision. I think this would need to be rebuilt from the ground up. It's probably using outdated APIs and I fear the data format coming out of it has changed considerably. Also, if you have problems with Custom Vison itself then I think you should look at Microsoft itself. I am sorry, this is not a simple upgrade. I haven't used Custom Vision itself for some time now, I would have to look into it again from the start.

[matthew-oleary02]

Thank you for getting back to me so quickly! I had a feeling that was the case because a few of the other MRTK tutorials from Microsoft have outdated instructions/API resources (one of them is supposed to help integrate the LUIS resource, an Azure resource that cannot be created anymore, for use on HoloLens2). It has been very confusing trying to figure out/test what exactly is wrong with the project, but this makes a lot more sense the way you put this into perspective on how old these tutorials are by this point. But would the goal of our project, using image recognition to identify a 3D Printer on HoloLens2, still be attainable using Microsoft Azure Custom Vision? Or should we integrate YoloV8 into our project instead since it's newer? My project partners and I are all beginners with Unity3D, HoloLens2, and Azure development so we are unsure of what is possible for making this project. If there is anything else I can specify about our project, please let me know.

[LocalJoost]

I think you have an attainable goal - as far as recognizing an object goes. I think both Custom Vision and Yolo are usable - I do know how to train Custom Vision (or I used to know), I really have no clue as how to create a Yolo model. I have recently made a demo that uses a YoloV7 model, but I got the model from somewhere else. You might want to have a look at it https://localjoost.github.io/HoloLens-AI-using-Yolo-ONNX-models-to-localize-objects-in-3D-space/ Bei aware of limitations. Yolo had to run on HoloLens. Custom Vision Models can run locally but you can also keep them in the cloud and use an API. This is typically more powerful, doesn't stress your HoloLens as much but is slightly slower.

Does this help?

[matthew-oleary02]

Yes, this helps a lot. I know how to train the custom vision model within a custom vision project after following one of Microsoft's tutorials but I am unsure as to how I can integrate this with Unity3D to be deployed onto HoloLens2 specifically. My project partners spoke with me today and told me that they spoke to you about integrating YoloV7 into their project. They're the ones making the project with using image recognition to identify pig hearts. We are all working for the same professor, but we are working on separate projects with mine focusing on image recognition to identify a Creality Ender 6 3D printer. If YoloV7 works better for image recognition since it has been tested to work on HoloLens2, then I would rather that be our focus since we have had many issues with Custom Vision. But, if there is a way of integrating our Custom Vision project into Unity3D to be deployed onto HoloLens2, then we could try doing that instead. Thank you!

[LocalJoost]

In ye olden days there was an API for Custom Vision models, but you could also download it as an ONNX model - it even came with some sample code. The first one I described here https://localjoost.github.io/using-azure-custom-vision-object/, the second one here https://localjoost.github.io/adapting-custom-vision-object/. But this is most likely VERY outdated

[matthew-oleary02]

The Custom Vision models do still have APIs but I cannot get them to work properly with Microsoft's tutorial the way they describe how to set the project up. I am going to take a look at your articles and see if I can still download it as an ONNX model like you describe. Thank you and I will get back to you after I test this out.

[matthew-oleary02]

I have an update and I think after following your instructions I think it can still work! I was able to create my own Custom Vision prediction model and export it as an ONNX file and it has a similar C# script. But, I am still running into a few errors. After going through this article, I get the three errors (highlighted in blue) and the rest come from when I tried to create a build in the build settings. Do you know what has to be fixed in order to get this to be able to be built? Here is a screenshot of what the errors look like in Unity3D and here is the ObjectDetection script I modified by following your article. If there is any more information you may need in order to be able to help me, please let me know I can let you know. Thank you!

[LocalJoost]

It is really hard to tell without a complete project and context. I think you might just be missing some usings? For the task error, I would expect you'd need to add

using System.Threading.Tasks;

SoftwareBitmapis a piece of UWP code, so it should probably be within #if !UNITY_EDITOR tags

[matthew-oleary02]

using System.Threading.Tasks; was inside the #if !UNITY_EDITOR tags so I moved it outside of those tags and then put SoftwareBitmap inside the #if !UNITY_EDITOR tags. Those errors have now been fixed! But, now I am getting another error that looks like this. I'm pretty sure this error is another reason why I cannot create a new build in the Unity3D build settings. I would like this error to be so that I could open this build (.sln file) in Visual Studio 2022 which can then be deployed onto HoloLens2. Hopefully, after fixing this error I could then create a build. Would you know how to fix this issue? Again, I have never used C# before so this is a bit new for me. Here is the revised code. Thank you!

[LocalJoost]

The declaration of probs is within #if !UNITY_EDITOR and the usage is not.

Listen... I don't want to be a Debbie downer, but you are apparently new to C#, I guess pretty new to Unity as well. You are trying to bang 5 year old code into something that compiles, I have no idea if the current CustomVision API even works, this was intended for HoloLens 1... you have to learn to crawl before you can walk, walk before you can run... and what you are trying to do is trying to fly. ;). Maybe it's time to do a step back. I don't think using this ancient code as a starting point is a good one

[matthew-oleary02]

So, where should I put the tag or get rid of it? Should I get rid of the tag around the declaration of probs or put a tag around the usage?

Well, that's why I want to see if I can get it to work. We would like image recognition (using whatever means necessary) working in our project. Whether that be through the use of Custom Vision or YoloV7 or V8. Do you think we should focus on YoloV7 or V8 instead if you think this won't work? Like I mentioned in a previous comment, you helped partners in my group with implementing YoloV8 in their project about image recognition with regards to pig's hearts. We have a quickly approaching deadline for this project and have been struggling for weeks trying to figure out/testing to see if this works (using that Microsoft tutorial i mentioned earlier) and I feel that we are the closest to getting it yet. This may be a bit too advanced/ambitious for what we've done so far, but I do really feel that we are close to getting it to work and I'm not going to give up just yet.

[LocalJoost]

TBH, I think it would be best to ask you partners to build a YoloV8 model, copy their project and use that. It's very difficult for me to see what is going wrong in a separate file, outside of a project, with only screenshots to work with. Also, even if the code runs, there is no way to know if it actually still works. And ATM I have not much time, next week even less. I am sorry