search

Unity-Technologies / com.unity.webrtc

WebRTC package for Unity
Other
753 stars 191 forks source link

[BUG]: Resizing the screen causes a a long delay with a frozen picture at the start of a stream, with periodic freezes thereafter on mobile #893

Open doctorpangloss opened 1 year ago

doctorpangloss commented 1 year ago

Not related to #752, which was definitely solved by the upgrade to M107 and resolves the keyframing issue.

Package version

reproduces: ad8c3776d9d3748d893a73fd2544cdbac231e96b reproduces: 3.0.0-pre4 does not reproduce: 2.4.0 (all versions)

Environment

* OS: Windows 2022
* Unity version: 2021.3.22f1 URP

Steps To Reproduce

  1. Use a scene with a simple button and NvCodec.
  2. Create an offer from the Unity scene.
  3. Connect a remote visitor.
  4. Try to mash the button. Observe it works.
  5. Resize the stream.
  6. Observe a 3s 10s freeze between receiving the first frame (a still picture) and the button being interactable.

The software codecs do not exhibit the issue.

Current Behavior

There is a long time between the stream starting to send picture versus the subsequent frames in the stream from the client's perspective being interactable.

Expected Behavior

No freezing.

Anything else?

Setting keyframe interval to a small number makes the scene interactable much sooner. Then, after a short period of time, switching back to infinite goplength works.

Unlike the previous issue, this hotfix no longer works since upgrading to M107.

Increasing kLimitBufferCount to 600 resolves the issue.

Reverting https://github.com/Unity-Technologies/com.unity.webrtc/commit/7481842b78a0ba55222df45d725afd0ec7fff139 resolves the issue.

No response

doctorpangloss commented 1 year ago

This issue was caused by... profiling.

doctorpangloss commented 1 year ago

Still reproduces. Just use the editor enough.

karasusan commented 1 year ago

In this changeset, native plugins are not updated. https://github.com/Unity-Technologies/com.unity.webrtc/commit/ad8c3776d9d3748d893a73fd2544cdbac231e96b

We are going to update native plugins when releasing the new package.

doctorpangloss commented 1 year ago

I built the plugin on that changeset.

I narrowed it down to requesting a change in streaming size. I am going to profile the Resize call. You will probably want to use the Unity-provides-the-input-resources-for-NVENC solution. Maybe from RTHandles, which tolerates changes in screen size very well. Otherwise, I think you can create 2048x2048 textures and specify a region in the encoder, rather than resizing the CUDA array every time. I'll follow up soon.

doctorpangloss commented 1 year ago

@karasusan This is caused by 7481842b . When doing things that cause the encoder to be reset, like resizing the stream, frames will be buffered and quickly exceed the kLimitBufferCount. Then "stale frames" are only released after 10s. That's why I observed 10s freezes. Increasing kLimitBufferCount resolves the issue.

There will be other circumstances where Unity will be able to queue frames to encode, but webrtc will be unable to encode them.

karasusan commented 1 year ago

@doctorpangloss Thanks for investigating the issue.

Currently, this keep frame buffers for one video resolution. To solve this issue, we need to prepare buffers for multiple resolutions.

kannan-xiao4 commented 1 year ago

memo: WRS-463

davide-aldegheri commented 1 year ago

Hi there, I don't know if it's related to this issue but I found a strange behaviour similar to this. Using the latest pre.4 version I've noticed that my streams were sometimes dropping frames because they weren't actually using any bandwidth at all, even checking with the editor utility I've noticed that the frames in upstream were dropping to 0. This doesn't happen with pre.3 which I've downgrade to as of now to avoid this inconvenience

karasusan commented 1 year ago

@davide-aldegheri Thanks for sharing the issue. What device are you using? And which graphics device?

davide-aldegheri commented 1 year ago

@karasusan I was using multiple devices, the one with the unity application conducting the stream was a PC with an NVidia 2070 on with the 531.68 drivers. The receivers were either browsers (Chrome) or other PCs (one without a dedicated GPU and one with an NVidia card). Also, I was using software encoding. Unity version is 2021.3.20f1.

EDIT: I've seen that pre.5 has been released so I've tested that and I can confirm that the issue is still there, maybe with a lower impact but as you can see this is the difference between pre.5 and pre.3:

pre.5 pre5 pre.3 pre3

with pre.4 I had longer and discontinued drops

karasusan commented 1 year ago

@davide-aldegheri Thanks for sharing the detail.