search

IntelRealSense / librealsense

Intel® RealSense™ SDK
https://www.intelrealsense.com/
Apache License 2.0
7.6k stars 4.83k forks source link

D435i SLAM #7844

Closed astaikos316 closed 3 years ago

astaikos316 commented 3 years ago
Required Info
Camera Model D435i
Firmware Version 05.12.08.200
Operating System & Version Windows 10
Platform PC
SDK Version Latest Unity Wrapper
Language Unity
Segment AR

Issue Description

I have been trying to run the various unity samples with my D435i and have run into a problem with the SLAM sample. Since i am trying to do some AR research, having SLAM ability is critical. When I run the scene in editor i get the following error:

ExternalException: rs2_pipeline_start_with_config(pipe:0000028B37CA8330, config:0000028B37CA7950) Rethrow as Exception: Couldn't resolve requests Intel.RealSense.ErrorMarshaler.MarshalNativeToManaged (System.IntPtr pNativeData) (at <6d5b8eabc0b341e3bb8bf3481e1594d0>:0) Intel.RealSense.Pipeline.Start (Intel.RealSense.Config cfg) (at <6d5b8eabc0b341e3bb8bf3481e1594d0>:0) RsDevice.OnEnable () (at Assets/RealSenseSDK2.0/Scripts/RsDevice.cs:71)

Any help would be greatly appreciated.

MartyG-RealSense commented 3 years ago

Hi @astaikos316 The SLAM.unity sample program of the RealSense Unity wrapper is only for the RealSense T265 Tracking Camera model, unfortunately.

RealSense does have a D435i SLAM tutorial for ROS.

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

A Unity interface for ROS is available for free in the Unity Asset Store if it is vital for your project to use Unity, though I am not aware of a case where it has been linked to the RealSense ROS wrapper.

https://assetstore.unity.com/packages/tools/physics/ros-107085

astaikos316 commented 3 years ago

I am not trying to use ROS for this. For AR scenario doing remote assist on a windows 2 in 1 tablet, having SLAM access in Unity with the D435i will allow for hologram placement.

MartyG-RealSense commented 3 years ago

Could you tell me more about how you plan to place holograms please? This will help to determine whether SLAM is needed (since SLAM is typically used when the camera is attached to something that is moving, such as a mobile robot).

astaikos316 commented 3 years ago

using a directional raycast. location of camera is necessary as the user holding the tablet can be moving.

MartyG-RealSense commented 3 years ago

Rather than using SLAM, a different system called pose may be usable. Pose can be used by robot arms with a D435 camera attached to work out how to reach, grip and handle objects, to give just one example of how pose data can be used. Pose is basically the position and rotational angle of the camera or of an observed object. The official RealSense blog has an article about grasping that illustrates this.

https://www.intelrealsense.com/robot-grasping/

In regard to how pose could be applied to a D435 in Unity: a couple of years ago there was an unofficial extension developed for the RealSense Unity wrapper that made use of Aruco image tags placed on an object to work out the pose (position and rotation) of that object.relative to the camera position by observing the image tag placed on the object.

https://github.com/ajingu/RealSense#aruco

Googling for aruco hologram reveals links to numerous hologram projects that have taken the Aruco detection approach.

MartyG-RealSense commented 3 years ago

Hi @astaikos316 Do you require further assistance with this case, please? Thanks!

astaikos316 commented 3 years ago

Using Aruco markers for equipment maintenance and troubleshooting is not practical in my case. Have been looking for a way to attach a camera system that supports SLAM to a Windows Tablet to provide AR capabilities. Since it can be done in ROS, is there any anticipation for it to work in Unity?

MartyG-RealSense commented 3 years ago

There are rugged industrial Windows tablets that have a built-in D410 or D415 camera. That may be a more efficient solution than attaching an external USB RealSense camera to a tablet and carrying it around with the tablet. Here are some examples of such tablets:

https://business.panasonic.co.uk/mobile-solutions/news-and-events/panasonic-introduces-a-fully-rugged-tablet-with-integrated-intel-realsense-3d-camera-to-capture

https://ruggedtablet.msi.com/product/NF21

https://www.officer.com/command-hq/technology/computers-software/tablets-handhelds/press-release/21027456/dt-research-the-dt301x-rugged-tablet-is-meant-for-high-accuracy-measurements

Alternatively, there is a free ROS wrapper for Unity created by Siemens, though I do not know of any cases where it has been used with RealSense.

https://assetstore.unity.com/packages/tools/physics/ros-107085 https://github.com/siemens/ros-sharp/wiki

If a Windows tablet is not an absolute requirement, there was a project for using a D435i with an Android-based Oculus Quest headset in Unity.

https://github.com/GeorgeAdamon/quest-realsense

The original Oculus Quest is no longer available but Oculus Quest 2, which supports Android 10, has recently been launched.

https://www.oculus.com/quest-2/ https://developer.oculus.com/blog/introducing-oculus-quest-2-the-next-generation-of-all-in-one-vr/

MartyG-RealSense commented 3 years ago

Intel also published a blog article yesterday about using RealSense for holographic display.

https://www.intelrealsense.com/looking-glass/

MartyG-RealSense commented 3 years ago

Hi @astaikos316 Do you require further assistance with this case, please? Thanks!

MartyG-RealSense commented 3 years ago

Case closed due to no further comments received.