Access research mode from a c# script

[autimator]

Hello, we have the SensorVisualization sample working and would like to use the IR and depth camera in our C# (unity) application. The Media Foundation contains code that seems to created for this purpose, with functions like: “MediaFrameSourceKind.Depth” there are also people mention that its possible: (https://stackoverflow.com/questions/59948317/mediafoundation-gives-four-cameras-into-one-single-frame)

Somehow we can only see the locatable camera, not the depth/IR camera using this method. Do you know if it is possible to acces the other camera's of the hololens2 in this way? Any help would be appreciated!

[dorinung]

The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.

[dorinung]

Another possibility is using COM interop in .NET: https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop . Visual studio has a C# Holographic DirectX11 App(Universal WIndows) template. This shows using COM interop in .NET with DX11 interfaces. Template instructions at:

https://docs.microsoft.com/en-us/windows/mixed-reality/develop/native/creating-a-holographic-directx-project

[autimator]

Thank you for the information. Over the last days I managed to deploy a simple C++ dll on the HoloLens. But I sadly don’t have the skills I need to create a dll for the research mode.

Do you know if there are plans to create an unity plugin?

[dakesse]

The current research mode APIs can not be used from c#. They are c++ only. I think it is possible to write a C++ unity plugin that would access the cameras using native c++.

Yes, you can do that. You need to wrap the needed API calls in accessible functions in your DLL, via __declspec(dllexport). In Unity you can import these with [DllImport("myDll.dll", EntryPoint = "-dll funtion name-")] and use them. I currently rebuild my DLL so that it fits our requirements. But generally you need a function to initialize the sensors and some to query the respective data. you can orientate on the SensorVisualization sample for that or have a look in the API doc

[cookieofcode]

I found two C++ Unity Plugins HoloLens2-Unity-ResearchModeStreamer and HoloLens2-ResearchMode-Unity that provide some features of the Research Mode using a DLL, they should work fine.

I tried enabling the research mode in C# and .NET using COM interop (https://docs.microsoft.com/en-us/dotnet/standard/native-interop/cominterop) as mentioned by @dorinung. Using the following snippets, I'm aquiring an IntPtr to the unmanaged ResearchModeSensorDevice:

IntPtr hrResearchMode = NativeAPI.LoadLibraryA("ResearchModeAPI");
if (hrResearchMode == IntPtr.Zero) throw new InvalidOperationException("The ResearchModeAPI library could not be loaded.");
IntPtr procAddress = NativeAPI.GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice");
if (procAddress == IntPtr.Zero) throw new InvalidOperationException("Procedure CreateResearchModeSensorDevice not found.");
CreateResearchModeSensorDevice pfnCreate = (CreateResearchModeSensorDevice) Marshal.GetDelegateForFunctionPointer(procAddress, typeof(CreateResearchModeSensorDevice));
Marshal.ThrowExceptionForHR(pfnCreate(out IntPtr device));

Guid guid = typeof(IResearchModeSensorDevicePerception).GUID;
int queryInterface = Marshal.QueryInterface(device, ref guid, out IntPtr devicePerception);
Marshal.ThrowExceptionForHR(queryInterface);

[UnmanagedFunctionPointer(CallingConvention.StdCall)]
delegate int CreateResearchModeSensorDevice([Out] out IntPtr sensorDevice);

static class NativeAPI
{
     [DllImport("kernel32.dll")]
     public static extern IntPtr LoadLibrary(string dllToLoad);

     [DllImport("kernel32.dll", CharSet = CharSet.Ansi, ExactSpelling = true)]
     public static extern IntPtr LoadLibraryA(string lpLibFileName);

     [DllImport("kernel32.dll")]
     public static extern IntPtr GetProcAddress(IntPtr hModule, string procedureName);

     [DllImport("kernel32.dll")]
     public static extern bool FreeLibrary(IntPtr hModule);
}

But I can't use Marshal.GetObjectForIUnknown(devicePerception) (API), which leads to an InvalidCastException. The HRESULT error code corresponds to 0x80004002 (E_NOINTERFACE). Using Marshal.GetTypedObjectForIUnknown(devicePerception, typeof(IResearchModeSensorDevicePerception) (API) leads to a MissingMethodException)

[ComImport]
[Guid("C1678F4B-ECB4-47A8-B6FA-97DBF4417DB2")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IResearchModeSensorDevicePerception
{
     [PreserveSig]
     int GetRigNodeId([Out] out Guid rigNodeId);
}

I took IResearchModeSensorDevicePerception instead of IResearchModeSensorDevice for simplicity, but tried the above with both interfaces. I'm stuck here, does anyone have an idea or hint?

[dakesse]

Had some issues with marshaling too and ended up using an unsafe block to circumvent them and just worked with pointers in C#:

...
 unsafe {
   // get data from dll
   SensorDataStruct* dataPointer = getAccDataBatched();

   // save it in struct
   for (int i = 0; i < ACCBATCHSIZE; i++) {
     accData.timestamp = dataPointer[i].VinylHupTicks;
     accData.sensorData = new Vector3(dataPointer[i].SensorValueX, dataPointer[i].SensorValueY, dataPointer[i].SensorValueZ);

     ...
     // fire event with new sensor data
     ...
   }
 }
...

The sruct on the plugin side looks like this:

extern "C" {
  //##### Structs #####
  struct SensorDataStruct {
    uint64_t VinylHupTicks;
    float SensorValueX;
    float SensorValueY;
    float SensorValueZ;
  };

  //##### Class #####
  class DllExports {
public:
    DllExport static SensorDataStruct* GetAccDataBatched();

    ...
  };
}

[JiangTao100]

@dakesse I built my own dll below, but when I imported the dll file in unity and deployed it to hololens2, there was something wrong . The program stopped at OpenAccSensorStream(). I 'm kind of confused where the error is.Could you help me? `

pragma once

include "pch.h"

include "ResearchModeAPI.h"

ifdef myResearchMode

define DllExport _declspec(dllexport)

else

define DllExport _declspec(dllimport)

endif

ifdef _cplusplus

extern "C" {

endif

struct  SensorDataStruct
{
    uint64_t VinylHupTicks;
    uint64_t SocTicks;
    float SensorValueX;
    float SensorValueY;
    float SensorValueZ;
    float temperature;
};
class DllExport DllExports
{
public:
        static void InitSensors();
        static void OpenAccSensorStream();

        static SensorDataStruct *GetAccDataBatched();

        static void CloseAccSensorStream();

        static void ReleaseSensors();

        static void ImuAccessOnComplete(ResearchModeSensorConsent consent);

};
    //##### Global Vars #####
    // static vars for research mode init
       static IResearchModeSensorDeviceConsent* m_pSensorDeviceConsent;
    static IResearchModeSensorFrame *pSensorFrame;

    static IResearchModeSensorDevice *m_pSensorDevice;
    static std::vector<ResearchModeSensorDescriptor> m_sensorDescriptors;       
    static IResearchModeAccelFrame *m_pSensorAccelFrame;

    // static vars for sensor access
    static IResearchModeSensor *m_pAcclSensor;

    HRESULT hr = S_OK;

ifdef _cplusplus

}

endif

` and the .cpp file like this

define myResearchMode

include "pch.h"

include "myResearchMode.h"

extern "C" HMODULE LoadLibraryA( LPCSTR lpLibFileName ); static ResearchModeSensorConsent imuAccessCheck;
static HANDLE imuConsentGiven;

_declspec(dllexport) void ImuAccessOnComplete(ResearchModeSensorConsent consent) { imuAccessCheck = consent; SetEvent(imuConsentGiven); }

_declspec(dllexport) void InitSensors()

{ size_t sensorCount = 0; imuConsentGiven = CreateEvent(nullptr, true, false, nullptr);

HMODULE hrResearchMode = LoadLibraryA("ResearchModeAPI");
if (hrResearchMode)
{

    typedef HRESULT(__cdecl* PFN_CREATEPROVIDER) (IResearchModeSensorDevice** ppSensorDevice);
    PFN_CREATEPROVIDER pfnCreate = reinterpret_cast<PFN_CREATEPROVIDER>(GetProcAddress(hrResearchMode, "CreateResearchModeSensorDevice"));
    if (pfnCreate)
    {
        hr = pfnCreate(&m_pSensorDevice);
    }
    else
    {
        hr = E_INVALIDARG;
    }

}
m_pSensorDevice->QueryInterface(IID_PPV_ARGS(&m_pSensorDeviceConsent));
m_pSensorDeviceConsent->RequestIMUAccessAsync(ImuAccessOnComplete);

m_pSensorDevice->DisableEyeSelection();

m_pSensorDevice->GetSensorCount(&sensorCount);
m_sensorDescriptors.resize(sensorCount);

m_pSensorDevice->GetSensorDescriptors(m_sensorDescriptors.data(), m_sensorDescriptors.size(), &sensorCount);

for (auto& sensorDescriptor : m_sensorDescriptors)
{
    if (sensorDescriptor.sensorType == IMU_ACCEL)
    {

        m_pSensorDevice->GetSensor(sensorDescriptor.sensorType, &m_pAcclSensor);
    }
}

}

_declspec(dllexport) void OpenAccSensorStream() {

m_pAcclSensor->GetNextBuffer(&pSensorFrame);
hr = pSensorFrame->QueryInterface(IID_PPV_ARGS(&m_pSensorAccelFrame));

}

_declspec(dllexport) SensorDataStruct* GetAccDataBatched() {

// get the sensor Frame and next buffer
    const AccelDataStruct* accelBuffer;
    size_t BufferOutLength;
    // read sensor values ...
    hr = m_pSensorAccelFrame->GetCalibratedAccelarationSamples(&accelBuffer, &BufferOutLength);

    SensorDataStruct* data = new SensorDataStruct[BufferOutLength];  // ?
    // … and copy them to return array
    for (UINT i = 0; i < BufferOutLength; i++)
    {
        data[i].VinylHupTicks = accelBuffer[i].VinylHupTicks;
        data[i].SocTicks = accelBuffer[i].SocTicks;
        data[i].SensorValueX = accelBuffer[i].AccelValues[0];
        data[i].SensorValueY = accelBuffer[i].AccelValues[1];
        data[i].SensorValueZ = accelBuffer[i].AccelValues[2];
        data[i].temperature = accelBuffer[i].temperature;
    }
    return data;

}

_declspec(dllexport) void CloseAccSensorStream() { m_pAcclSensor->CloseStream(); }

_declspec(dllexport) void ReleaseSensors() {

m_pSensorAccelFrame->Release();

}

[dakesse]

Sorry for my very late response. As far as I could see the difference between our approaches is that you export the whole class into the DLL. I can remember that I had some issues with that, so I exported every element/function of the class instead the class itself. So:

class DllExports {
  public:
    DllExport static void InitSensors();
    DllExport static void OpenAccSensorStream();
    DllExport static void GetAccDataStamped(float& accX, float& accY, float& accZ, UINT64& timestamp);
    DllExport static void CloseAccSensorStream();
    ....
    DllExport static void ReleaseSensors();
};

instead of:

class DllExport DllExports {
  public:
    static void InitSensors();
    static void OpenAccSensorStream();
    static SensorDataStruct *GetAccDataBatched();
    static void CloseAccSensorStream();
    static void ReleaseSensors();
    static void ImuAccessOnComplete(ResearchModeSensorConsent consent);
};