Roadmap

[gespona]

Why?

Sharing the roadmap with everyone is always good for transparency and best way to drive the development in order to bring the most value in each step.

What?

All feedback is more than welcome ! And we'd love to know more about you and get some insights:

• Main platform are you using? (MacOS, Win, Linux)
• Could you explain bit more about your use case? What are you looking to build with the Unity plugin?
• What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)
• Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

Features and Roadmap:

v1.0.0 Initial Release to AssetStore (free):

• No code approach. Just drag and drop some prefabs. Bullet-proof.
• Access to OAK device streams:
  • RGB and depth (point cloud viz currently ongoing dev).
    • About point cloud: Add support for external libraries: PCL, ... and VFX integration.
  • Access to IMU if available and retrieve system information.
  • Record and Replay capability.
• OAK Device Manager and multiple OAK devices support.
• OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family):
  • Face Detector, 3D Head Pose Estimation, Face emotion, Body Pose (MoveNet), Hand Tracking and Gesture recognition (Blaze Hands), Object detector (Tiny Yolo v3/4/5 and MobileNet SSD). DeepLabv3 segmentation. Background removal, Depth (only OAK-D family) - point cloud visualization
  • Multiple camera demo.
  • Example how to integrate your own pipeline
  • Integration with some paid assets (demos)
• OAK For CV/AI/Robotics: Unity Virtual OAK Camera

Version 1.0 is just the top of the iceberg. Foundations to build applications on top. Beta testing along the path to get v1.0 ready for assetstore.

Some topics for next versions (big blocks without much detail)

• More complex demos on top of pretrained models (see list below)
• Send unity virtual camera images to OAK pipeline.

Do you miss some models / use cases / integration with some specific paid asset? Please let us know.

• Android support. Integration with AR.
• End-to-end workflows for synthetic dataset generation, training and deploy.
• Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK. Publish to ROS.
• Integration with Unity Simulation and Unity Simulation Pro.
• Intergration with Unity ML-Agents.
• Define custom pipelines inside Unity (visual tool).
• Implement partial C# wrapper

How?

We're opening beta testing on the repo. The plan is start delivering all the features gradually following initial order and always covering full feature (unity plugin lib + unity project/demos) and for all platforms (Win, Mac and Linux)

Following features (v1.0) are ready:

• Device Manager
• Demo menu scene
• Demos:
• Device streams (rgb, depth), Point cloud visualization using VFX
• Object detector demo
• Face detector demo
• Body pose demo
• Head pose demo
• Face emotion demo
• Hand tracking demo

Please tell us about your project !

More resources: unity forums: https://forum.unity.com/threads/oak-for-unity-spatial-ai-meets-the-power-of-unity.1205764/ youtube unity plugin playlist: https://youtu.be/CSFOZLBV2RA?list=PLFzqMMJPSNSbsHp7QeJpOHrZu_1BAdDms

[gespona]

• Camera type is oak-D
• Yes, use USB3.0
• Time is not fixed，10 minutes or 1 hour
• I didn't understand what you meant
• My pc have 8GB memory, this program only use 300MB memory and I know how to cause the crash, just unplug the USB cable，I check this usb cable，there is no poor contact, it is inserted quite tightly, so I think this dosn't the main problem.

@jacoos

Do you mean after 10 minutes or 1 hour you unplug the usb cable? Unplug usb cable for sure produce crash as camera is communicating with host. I understand the crash is happening just running the program right?

• I didn't understand what you meant

I mean the crash is happening with build binary, not reproduce inside editor (?)

[gespona]

@jacoos btw could you open the issue and attach editor / player log ?

[garrett-tuer]

Hello, I just wanted to express my team's interest in this asset.

Main platform are you using? (MacOS, Win, Linux)

We primarily operate on Windows.

Could you explain bit more about your use case? What are you looking to build with the Unity plugin?

We are biomechanics researchers using full-body tracking and pose estimation in different apps/games with built-in biomechanics analysis. Historically, this type of work has been restricted to pre-recorded motions, but pose estimation software allows us to analyze movements in (near) real-time. Access to the camera's RGB and depth data as well as the pose estimation results (potentially converted to a consistent/universal format, regardless of the pose tool) and root (pelvis) position in world space, would be our primary needs. Ease of switching between different pose estimation software (MediaPipe, MMPose, etc.) and custom models within those software would also be very desirable for us. The ability to detect multiple people (incorporating YOLOPose? MMPose does this by default) would also be a major plus.

What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)

Likely OAK for Developers since, as mentioned above, we would like to be able to use custom pretrained models and multiple different pose estimation software.

Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

Our experience is very minimal. Our only use of the OAK API/SDK has been through an implementation of geaxgx's depth_blazepose repo (https://github.com/geaxgx/depthai_blazepose) with the current OAKForUnity code to stream a video feed and corresponding BlazePose results to a Unity scene in (near) real-time. We have not yet written any of our own code that interfaces with the API/SDK. This repo mostly handle our needs, but removing the need for the Unity bridge and adding any of the features mentioned above that are missing would be a major improvement.

[gespona]

Hello, I just wanted to express my team's interest in this asset.

Main platform are you using? (MacOS, Win, Linux)

We primarily operate on Windows.

Could you explain bit more about your use case? What are you looking to build with the Unity plugin?

We are biomechanics researchers using full-body tracking and pose estimation in different apps/games with built-in biomechanics analysis. Historically, this type of work has been restricted to pre-recorded motions, but pose estimation software allows us to analyze movements in (near) real-time. Access to the camera's RGB and depth data as well as the pose estimation results (potentially converted to a consistent/universal format, regardless of the pose tool) and root (pelvis) position in world space, would be our primary needs. Ease of switching between different pose estimation software (MediaPipe, MMPose, etc.) and custom models within those software would also be very desirable for us. The ability to detect multiple people (incorporating YOLOPose? MMPose does this by default) would also be a major plus.

What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)

Likely OAK for Developers since, as mentioned above, we would like to be able to use custom pretrained models and multiple different pose estimation software.

Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

Our experience is very minimal. Our only use of the OAK API/SDK has been through an implementation of geaxgx's depth_blazepose repo (https://github.com/geaxgx/depthai_blazepose) with the current OAKForUnity code to stream a video feed and corresponding BlazePose results to a Unity scene in (near) real-time. We have not yet written any of our own code that interfaces with the API/SDK. This repo mostly handle our needs, but removing the need for the Unity bridge and adding any of the features mentioned above that are missing would be a major improvement.

Hi @garrett-tuer ! Thank you very much for you feedback and interest in OAK For Unity. Evaluating different pose estimation models including multi-pose models makes sense. As probably you read above in previous comments, we're looking on some of those topics, taking into account RVC2 capabilities.