Inconsistent Accel stream in Unity C#

[thomaskole]

---

Required Info
Camera Model	D455
Firmware Version	05.13.00.55
Operating System & Version	WIN10
Platform	PC
SDK Version	v2.54.2 }
Language	C# Unity
Segment	others

Issue Description

When using the latest Realsense SDK in Unity, I can't get the Accel data behaving correctly. I have the following snippet:

[SerializeField] RsDevice rs;
    void Start()
    {
        rs.OnNewSample += Rs_OnNewSample;
    }

    private void Rs_OnNewSample(Frame f)
    {
        if (f.IsComposite)
        {
            using (var fs = f.As<FrameSet>())
            using (var poseFrame = fs.FirstOrDefault(Stream.Accel, Format.MotionXyz32f))
                if (poseFrame != null)
                    ProcessGyro(poseFrame);
        }
        else
        {
            using (var p = f.Profile)
                if (p.Stream == Stream.Accel && p.Format == Format.MotionXyz32f)
                    ProcessGyro(f);
        }
    }

    void ProcessGyro(Frame f)
    {
        var x = f.As<MotionFrame>();
        Vector3 v = new Vector3(x.MotionData.x, x.MotionData.y, x.MotionData.z);
        Debug.Log(v);

    }

To listen to Accel data. This is the RSDevice:

What happens is that when the device is enabled, for the first 1-5 seconds, data streams in, and then it keeps outputting the same value. So, new samples do come in via rs.OnNewSample, but the value that's put out is stuck.

The depth and IR work just fine. In RealsenseViewer, it keeps working too.

No errors or warnings show up in Unity. What could be the problem here?

[MartyG-RealSense]

Hi @thomaskole Does the problem still occur if you set a lower Accel framerate, please? D455 cameras manufactured before mid 2022 can be set to '63' as a minimum framerate, whilst D455 manufactured after mid 2022 do not support 63 and instead have '100' as their minimum.

If you are only using one infrared stream (the left one by default) then you should not need to set an index of '1' for Infrared and can leave its index as '0'.

The recommended camera firmware version for SDK 2.54.2 is 5.15.1.0. You can update to this version in the 2.54.2 version of the RealSense Viewer. The 5.13.0.55 firmware that you currently have installed is the recommended version for the older SDK versions 2.50.0 and 2.51.1.

[thomaskole]

Just updated the firmware. Same issue with framerate at 100. setting streamindex of the IR stream to "0" gives an ExternalException: rs2_pipeline_start_with_config

[thomaskole]

I've tried importing just the Intel.RealSense.unitypackage into an empty Unity project. The Accel data comes in it weird intervals. It will update for about half a second, and then "hang" for half a second, on repeat.

[MartyG-RealSense]

Is the Intel.RealSense.unitypackage file that you imported the one from the 2.54.2 assets list, please?

https://github.com/IntelRealSense/librealsense/releases/tag/v2.54.2

[thomaskole]

yes, it is.

[MartyG-RealSense]

Does IMU work normally on its own if the depth and infrared profiles are not used?

There is a known phenomenon where a program can work fine without IMU, or with IMU only, but once IMU and other types of stream (such as depth, color and IMU) are used simultaneously then problems can occur. A solution to this is to use callbacks, like in the C# scripting at https://github.com/IntelRealSense/librealsense/issues/11111

[thomaskole]

If I use a callback:

public class RSGyro : MonoBehaviour
{
    void OnEnable()
    {
        var pipeline = new Pipeline();

        Config cfg = new Config();

        var ctx = new Context();
        var devices = ctx.QueryDevices();
        var dev = devices[0];
        var sensors = dev.QuerySensors();
        var motionSensor = sensors[2];

        var gyroProfile = motionSensor.StreamProfiles
                    .Where(p => p.Stream == Stream.Accel)
                    .OrderBy(p => p.Framerate)
                    .Select(p => p.As<MotionStreamProfile>()).First();

        cfg.EnableStream(Stream.Accel, gyroProfile.Format, gyroProfile.Framerate);
        var ActiveProfile = pipeline.Start(cfg, f =>
        {
            ProcessAccelFrame(f);
        });

    }

    private void ProcessAccelFrame(Frame f)
    {
        if (f.IsComposite)
        {
            using (var fs = f.As<FrameSet>())
            using (var poseFrame = fs.FirstOrDefault(Stream.Accel, Format.MotionXyz32f))
                if (poseFrame != null)
                    PrintAccel(poseFrame);
        }
        else
        {
            using (var p = f.Profile)
                if (p.Stream == Stream.Accel && p.Format == Format.MotionXyz32f)
                    PrintAccel(f);
        }
    }

    void PrintAccel(Frame f)
    {
        var x = f.As<MotionFrame>();
        Vector3 v = new Vector3(x.MotionData.x, x.MotionData.y, x.MotionData.z);
        Debug.Log(v);
    }

}

I receive exactly 16 samples on start, then nothing:

It does not matter if the RsDevice component is enabled.

[thomaskole]

If I disable and re-enable the "RSGyro" script, I get new samples. I can do this a few times, then unity crashes silently without crash report.

[MartyG-RealSense]

One RealSense user at https://github.com/IntelRealSense/librealsense/issues/9278#issuecomment-879423474 took the approach of removing all profiles from RsDevice and said that this caused all 4 stream types (depth, color, infrared, IMU) to be enabled.

What happens to your RSGyro script if there are no profiles listed in RsDevice?

[thomaskole]

As stated above:

It does not matter if the RsDevice component is enabled.

Even with just the gyro script, and no "RsDevice" component (either disabled or not in the scene at all), I get the behavior as explained above.

[MartyG-RealSense]

It appears that you are disabling RsDevice (which handles the enabling and streaming of RealSense stream profiles) and having your RSGyro script define the streams and start the pipeline instead.

It might be better to insert your ProcessAccelFrame void into the RsDevice script file and let RsDevice handle the camera control via the defined profiles.

[MartyG-RealSense]

Actually, it looks as though once OnEnable calls the ProcessAccelFrame void, the contents of ProcessAccelFrame would only run once and then stop because it would be a one-shot void instead of a looping one like Update(). So if you want ProcessAccelFrame to loop and generate results continuously then you might want to add ProcessAccelFrame(f); to the end of it so that it loops back to the start of the void and runs the instructions again.

Correction: it looks as though ProcessAccelFrame runs once as a one-shot and then jumps to the ProcessGyro void. But I see no way for ProcessGyro to loop or for ProcessAccelFrame to run more than once.

[thomaskole]

Is that not what the pipeline.Start is for? Should the callback not be responsible for the loop? Should I call pipeline.Start multiple times?

When the ProcessAccelFrame function is called as a callback, I see that is called outside of the Unity main thread, which seams to imply that it works - but only 16 times.

ProcessAccelFrame and thus ProcessGyro happens more than once in my example. In fact, it happens 16 times. Excuse the processgyro misnomer - I will edit my original comment to have a better name for it - in this case PrintAccel

[MartyG-RealSense]

pipeline.start will start the camera and publish the streams. It should only be called once when the script is run and not be within a loop. Once it is called, the streams will be continuously active until pipeline.stop is called.

The streams will be continuously active. But ProcessGyro will be non-looping and only be run each time that it is called. So if ProcessGyro was called 16 times then it would only run the Vector3 instruction 16 times, one for each call of the function name.

[thomaskole]

I think there is a misunderstanding.

ProcessAccelFrame(f) is called in the callback of the pipeline.Start.

var ActiveProfile = pipeline.Start(cfg, f =>
        {
            ProcessAccelFrame(f);
        });

Am I wrong to assume that this callback (pipeline.Start(cfg, f => {my code here}) is called for every new sample, up until pipeline.Stop() is called?

What's happening now is that this callback is called 16 times, not by me but by the realsense stream. After 16 samples, there are no further updates.

[MartyG-RealSense]

If a RealSense script uses callbacks then typically the word callback will be inserted in the brackets of the pipeline start line so that the script knows that it is a callback script. For example:

pipeline.Start(cfg, callback)

My knowledge of callback programming (and also advanced Unity programming) is admittedly limited though, unfortunately.

[thomaskole]

I've done a bit more testing, and I think I'm getting closer to the solution:

Let's focus only on the callback:

        pipeline.Start(cfg, f =>
        {
            Debug.Log(f.Profile.Format);
        });

This works. The output of f.Profile.Format (MotionXyz32f) is printed for as long as I leave the game running.

Introducing the As<MotionFrame> seams to crash the stream, with no errors.

        pipeline.Start(cfg, f =>
        {
            var mf = f.As<MotionFrame>();
            Debug.Log(mf);
        });

This gives the result as before - 16 updates, then nothing.

I tried a marshall operation, like suggested here: https://github.com/IntelRealSense/librealsense/issues/11111#issuecomment-1318875500

        pipeline.Start(cfg, f =>
        {
            Vector3 v = Marshal.PtrToStructure<Vector3>(f.Data);
            Debug.Log(v);
        });

This seems to work (though more testing is needed). It seems that .As<MotionFrame> is broken in the SDK, and should be fixed.

[MartyG-RealSense]

It's great to hear that you made very significant progress. I look forward to a further update after your testing.

[thomaskole]

After a few seconds the editor crashes. This is what's found in the Editor log:

=================================================================
    Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

So, it seems pretty broken.

[MartyG-RealSense]

That error message does not seem to have occurred before in relation to RealSense. Googling for it indicates that it occurs in non-RealSense projects too. Here is an example case.

https://discussions.unity.com/t/unity-crashes-on-startup-got-a-unknown-while-executing-native-code/253054

[thomaskole]

I think the implication here is that the fault can be inside a DLL, and I suspect the realsense DLL to be at fault. When I don't enable the accelarometer, there is no crash.

[MartyG-RealSense]

There is another example of C# marshal code at https://github.com/IntelRealSense/librealsense/issues/2996#issuecomment-451068939 which was provided shortly before MotionFrame support was added to the SDK.

[thomaskole]

I'm using an identical marshal operation.

My marhsal: Vector3 v = Marshal.PtrToStructure<Vector3>(f.Data); vs theirs: var b = Marshal.PtrToStructure<Vector3>(accelFrame.Data);

But instead of using WaitForFrames(), I'm using a callback, as per your comment earlier: https://github.com/IntelRealSense/librealsense/issues/12250#issuecomment-1748692556

If I don't use a callback but WaitForFrames() instead, just like the example from your comment, I get back to the original situation: 16 received frames, then nothing.

[MartyG-RealSense]

So the current situation is that the program works fine if marshal code is used instead of MotionFrame?

[thomaskole]

No, the current situation is that when a marshall is used, the application crashes after a few seconds. I've tested this on two PC's now, same result.

[MartyG-RealSense]

As a problem with MotionFrame was also previously reported in https://github.com/IntelRealSense/librealsense/issues/11111 I will highlight the issue to my Intel RealSense colleagues.

However, if a bug in MotionFrame is confirmed then it would not immediately resolve your issue unfortunately due to the time required to develop a fix and then release it in a future version of the RealSense SDK.

[thomaskole]

Thanks for confirming and relaying the information.

Right now, this bug forms a serious problem in our production, as we are not able to use the IMU at all. Since this problem is already known for about a year, can we expect a fix somewhere soon? We need to know if we need to spend any more development resources on this, or if we need to abandon IMU-related features.

Thanks again for your time!

[MartyG-RealSense]

You are very welcome. Thanks very much for your patience. I have communicated the urgency of the issue to my colleagues and will let you know as soon as I receive feedback.

[MartyG-RealSense]

After consulting with my colleagues, an official internal Intel bug report has been created so that this issue can be investigated further by the RealSense team, and I have added an Enhancement label to this issue to signify that it should be kept open.

[thomaskole]

Hi, are there any updates on this issue?

[MartyG-RealSense]

Hi @thomaskole An official Intel internal bug report was created about this issue so that it can be investigated by the RealSense team, but there is no new information to report at this time unfortunately.

[MartyG-RealSense]

Hi @thomaskole The current latest SDK version 2.55.1 made some changes to the Accel stream of the IMU.

Are you able to test 2.55.1 to check whether the changes make a difference to the MotionFrame() issue in Unity, please?

[thomaskole]

It seems to work. I have upgraded the realsense to firmware version 5.16.0.1, and the unitypackage to 2.55.1. Using the code from the first post here results in a steady stream.

We will test this a little further but so far, so good!

[MartyG-RealSense]

That's great to hear. I look forward to your next report after tests. Good luck! :)

[MartyG-RealSense]

Hi @thomaskole Do you have an update about this case that you can provide, please? Thanks!