search

RenderHeads / UnityPlugin-AVProLiveCamera

AVPro Live Camera is a Unity plugin for webcams, TV cards and capture cards
https://renderheads.com/products/avpro-live-camera/
11 stars 0 forks source link

Timekeeping broken/Dropped frames not reported #33

Open BlueprintBen opened 3 years ago

BlueprintBen commented 3 years ago

Bug Description GetLastFrame/GetLastFrameUploaded doesn't increase at the rate predicted by CurrentFramerate, and GetCaptureDroppedFrames doesn't report any dropped frames during testing despite CaptureFPS dropping below CurrentFramerate. This significantly limits our ability to accurately perform timekeeping on inputs.

My Setup:

To Reproduce

  1. Create a Unity scene with an AVProLiveCamera component configured to start using a test webcam on awake.
  2. Add the attached TimingTest.cs component to the same GameObject (ensure correct file extension)
  3. Run the scene and wait 10 seconds for TimingTest to finish (it will stop the application itself)
  4. Review the console/log to see the unaccounted-for discrepancy between time elapsed and (frames seen / framerate).

Example Results image

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

BlueprintBen commented 3 years ago

Any progress on this issue? Not being able to keep track of the 'real' index of a frame drastically reduces the utility of this plugin for us unfortunately.

kahnivore commented 3 years ago

Hi @BlueprintBen

Sorry for the slow response on this.

Basically we need to investigate this further to see if this is an internal DirectShow issue or something to do with our plugin implementation, the latter which we could then look into improving.

We will look into it ASAP just haven't had anyone available to do so.

Thank you for your patience.

BlueprintBen commented 3 years ago

I've tested as far back as 2.7.5 without any change, so now imagining that this is more structural. Is there a device timestamp for each frame that could at least be acquired to calculate an approximate 'realtime' frame index? Otherwise we're stuck using when we see the new frame is available, which introduces a bunch of variability/latency.

kahnivore commented 3 years ago

We could probably expose a device timestamp quite easily, if this would work for you?

The underlying issue being that even if we expose the timestamp, it's still tricky to know if the frame you have is the frame you have the timestamp for - addressing this issue requires a more fundamental fix that we need to spend some proper time on.

Let us know if simply exposing the timestamp could help you.

BlueprintBen commented 3 years ago

Sorry, just a call like Plugin.GetDeviceTimestamp() isn't going to help, since it is really more data about the specific frame being accessed that I need. If the timestamp was for the active frame, at least I could calculate if it's 33ms higher than the last one (= no drops) or more (= drops), for example and make up an index from that.

Can you explain what the GetCaptureDroppedFrames() function does report since I might still be missing some aspect of the timing.

kahnivore commented 3 years ago

The timestamp would be for the active frame. We can add this in for you.

BlueprintBen commented 3 years ago

OK thanks! Definitely a leg up on the timing in cases where we're trying to match things up precisely. Let know know if you want me to try out a build.

kahnivore commented 3 years ago

We had a look into this and it doesn't seem as straightforward as anticipated.

We'll have to see if we have any luck with this in the future, otherwise it might be something for our next major version.

BlueprintBen commented 3 years ago

OK well thanks for investigating. Glad to have at least surfaced the value in having this type of information available.

BlueprintBen commented 2 years ago

Have come back to this and am at a bit of a loss as to why it's not feasible to implement passing along the timestamp that each DirectShow MediaSample (as long as not preview pin) has per the DirectShow docs. Feels like a core component of DirectShow and functionality that will really allow this plugin to sync with Unity better.

That page also mentions accessing Media Times which seem even more appropriate for my purposes. Anyway would love any more insight into the blockers if possible.

Thanks!