Variable Frame Rate for Software_device

[stephen-systemfriend]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D400
Operating System & Version	Win 10
Platform	PC
SDK Version	2.20
Language	unity
Segment	AR

Issue Description

For my use case (streaming the realsense data from a remote source across the internet then displaying it using the realsense tools), I will have a variable framerate for the input. However, the Matcher is very sensitive to framerate changes, and drops large quanties of whichever frame I add to the syncer first. Is there any way to fix this?

Also, sometime onNewSample is Null even when I have the rest of the pipeline attached to it.

here's my test code for reference.

`using System.Collections; using System.Collections.Generic; using UnityEngine; using Intel.RealSense; using System;

public class RSSoftwareDevTest : RsFrameProvider {

public float[] intrinsicsCoeffics = new float[] { 0, 0, 0, 0, 0 };

//the device by which we generate our frames
private SoftwareDevice framegenerator;
private SoftwareSensor colorGenerator, depthGenerator;
private VideoStreamProfile colorStream, depthStream;
private Syncer sync;
private int imgWidth, imgHeight;

public int fps = 30;

void Awake()
{
    framegenerator = new SoftwareDevice();
    depthGenerator = framegenerator.AddSensor("Depth");
    colorGenerator = framegenerator.AddSensor("Color");
    framegenerator.SetMatcher(Matchers.Default);
    Application.targetFrameRate = fps;
}

//this is a frame provider
/// <summary>
/// Notifies upon streaming start
/// </summary>
public override event Action<PipelineProfile> OnStart;

/// <summary>
/// Notifies when streaming has stopped
/// </summary>
public override event Action OnStop;

/// <summary>
/// Fired when a new frame is available
/// </summary>
public override event Action<Frame> OnNewSample;

public Texture2D colorTex;
public string depthTexPath;
int frameNumber = 0;
private ushort[] depthData;

// Update is called once per frame
void Update()
{
    if (colorTex.width != imgWidth || colorTex.height != imgHeight)
    {
        depthData = new ushort[colorTex.width * colorTex.height];
        if (depthTexPath != null)
            try
            {
                UnityEngine.Windows.File.ReadAllBytes(depthTexPath).CopyTo(depthData, 0);
            }
            catch (System.Exception e)
            {
                Debug.LogError(e);
            }
        var intrinsics = new Intrinsics
        {
            width = colorTex.width,
            height = colorTex.height,
            ppx = imgWidth / 2,
            ppy = imgHeight / 2,
            fx = colorTex.width,
            fy = colorTex.height,
            model = Distortion.BrownConrady,
            coeffs = intrinsicsCoeffics
        };

        var colorProfile = new SoftwareVideoStream
        {
            type = Stream.Color,
            index = 0,
            uid = 101,
            width = colorTex.width,
            height = colorTex.height,
            fps = fps,
            bpp = 3,
            format = Format.Rgb8,
            intrinsics = intrinsics
        };

        colorStream = colorGenerator.AddVideoStream(colorProfile);

        var dpthIntrinsics = new Intrinsics
        {
            width = colorTex.width,
            height = colorTex.height,
            ppx = colorTex.width / 2,
            ppy = colorTex.height / 2,
            fx = colorTex.width,
            fy = colorTex.height,
            model = Distortion.BrownConrady,
            coeffs = intrinsicsCoeffics
        };

        var depthProfile = new SoftwareVideoStream
        {
            type = Stream.Depth,
            index = 0,
            uid = 100,
            width = colorTex.width,
            height = colorTex.height,
            fps = fps,
            bpp = 2,
            format = Format.Z16,
            intrinsics = dpthIntrinsics
        };

        depthStream = depthGenerator.AddVideoStream(depthProfile);
        //depthGenerator.AddReadOnlyOption(Option.DepthUnits, 1.0f / 5000);

        depthGenerator.Open(depthStream);
        colorGenerator.Open(colorStream);
        sync = new Syncer();
        colorGenerator.Start(sync.SubmitFrame);
        depthGenerator.Start(sync.SubmitFrame);

        var extr = new Extrinsics();
        extr.rotation = new float[] { 1, 0, 0, 0, 1, 0, 0, 0, 1 };
        extr.translation = new float[] { 0, 0, 0 };

        depthStream.RegisterExtrinsicsTo(colorStream, extr);
          //different frame shape therefore we have to setup the streams
        imgHeight = colorTex.height;
        imgWidth = colorTex.width;
    }

    //update our position
    if (sync != null)
    {
        colorGenerator.AddVideoFrame<byte>(colorTex.GetRawTextureData(), colorTex.width * 3,
            3, System.DateTime.Now.Millisecond, TimestampDomain.HardwareClock, frameNumber, colorStream);
        depthGenerator.AddVideoFrame<ushort>(depthData, colorTex.width * 2,
           2, System.DateTime.Now.Millisecond, TimestampDomain.HardwareClock, frameNumber, depthStream);

        using (var frameset = sync.WaitForFrames())
        {

            var depthframe = frameset.FirstOrDefault(Stream.Depth);
            var colorframe = frameset.FirstOrDefault(Stream.Color);

            if (colorframe == null) Debug.LogWarning("colorframe was null");

            if (depthframe == null) Debug.LogWarning("depthframe was null");

            //send it using the RS pipeline
            var onNewSample = OnNewSample;
            if (onNewSample != null)
            {
                onNewSample(frameset.AsFrame());
            }
            else
            {
                Debug.LogWarning("OnNewSample is null");
            }
        }
        frameNumber++;
    }
}

void OnEnable()
{
    if (OnStart != null)
        OnStart(ActiveProfile);
}
void OnDisable()
{
    OnNewSample = null;

    if (Streaming && OnStop != null)
        OnStop();

    if (ActiveProfile != null)
    {
        ActiveProfile.Dispose();
        ActiveProfile = null;
    }

    Streaming = false;
}
void OnDestroy()
{
    // OnStart = null;
    OnStop = null;

    if (ActiveProfile != null)
    {
        ActiveProfile.Dispose();
        ActiveProfile = null;
    }
}

} `

[dorodnic]

Hi @stephen-systemfriend What specifically is causing you problems? The initial frame drops before syncer picks up on both streams, or its sensitivity to the advertised FPS? The later is used for frame prediction, to reduce latency in some scenarios.

[stephen-systemfriend]

My expected result from running this script from an object in unity would be that it maybe drops a couple of frames at the start, then proceeds to producing one frame per update pretty consistently. Instead I find that roughly half of the time, it instead spits out the warning "colorframe was null". Im not certain why this happens, but I think its due to the system's sensitivity to the advertised FPS.

Perhaps it would be good if there was a variant of the Syncer or the Matcher (whichever it is) that is agnostic to the framerate specifically for use with software sources?

[RealSenseCustomerSupport]

---

@stephen-systemfriend Sorry for late response. Is this issue still existing for you? Looking forward to your reply. Thanks!

[RealSenseCustomerSupport]

---

@stephen-systemfrien Any update? Thanks!

[stephen-systemfriend]

Sorry, it's been really busy with other work, so I haven't been able to confirm, but as far as I can tell, it still drops a lot of frames once I connect it to the network device instead. I'm expecting degraded performance due to the latency, but I could do without it also halving the effective frame rate again by dropping a whole load more of the frames.

[RealSenseCustomerSupport]

---

@stephen-systemfriend Any chance to confirm? And any other questions? Looking forward to your update.

[RealSenseCustomerSupport]

---

@stephen-systemfriend Any update? Thanks!

[RealSenseCustomerSupport]

---

@stephen-systemfriend As we didn't get response from you for weeks, will close this at the point. Please feel free to create another one if you still have questions or issues. Thanks!