search

saagarjha / Ensemble

Cast Mac windows to visionOS
https://testflight.apple.com/join/Pq1HzHqe
GNU Affero General Public License v3.0
847 stars 37 forks source link

Frames dropping with action #31

Closed xplorer256 closed 3 months ago

xplorer256 commented 4 months ago

There is a strange behaviour with resizing windows on visionOS.

A new window starts with a high frame rate as expected. Even videos and TV are transmitted with high frame rate at initialisation. As soon you try to resize a window on visionOS, the frame rate suddenly drops down to 1 per second and this problem remains whatever you try. New (possibly additional) windows are created with normal frame rate again. Until you resize it.

The 1 second frame update makes it somehow useless. But moving the window without resizing does work. Only resizing does cause the issue.

So it seems there is a problem with some of the code at TCP transport level or encoder level or the screen capture. But I think it is something on the visionOS side, that causes a drop of frames and then refreshes with merely 1 frame per second. Resizing a window probably interrupts something inside the signal flow permanently.

This is quite an important issue and I do not think, it is on the macOS side, as newly created windows always are running fine again.

If you are interested, I could make a video.

saagarjha commented 4 months ago

A video would be quite helpful :)

xplorer256 commented 4 months ago

Can you not replicate this issue? Is it specific to me? I do not think that it is specific, as i downloaded several versions (including the latest release) and it always has the same behaviour.

But I will try to capture a video.

xplorer256 commented 4 months ago

https://github.com/saagarjha/Ensemble/assets/159421565/934bed10-8431-4e0e-87e1-33b38773041f

Here's the video, capturing a TV app stream on the mac. Despite that there is a temporary audio dropout om the source at the beginning, there is this permanent issue after resizing windows. It is easily reconstructable each time. I used the latest commit and compiled it for this demonstration.

This happens with any kind of window, also with usual (more static) app windows without requiring such high video frame update rates. But such app windows are useless with a frame rate of 1 second after resizing nevertheless.

So this issue is not quite ignorable.

ps: My WLAN connection is fast enough to stream 4K videos and doing HQ local screen sharing between macs without any issue. I have a setup of 2 external monitors connected to my MacBook Pro with M2 Pro chip. Ensemble is able to capture the windows on all monitors.

saagarjha commented 4 months ago

It doesn't do that for me, no. I do know of some cases where the video degrades and I see symptoms like the ones you are describing, but I haven't been able to them to window resizing, so I am very curious to see if I am missing something.

xplorer256 commented 4 months ago

ps: I think it is an out of sync issue, that is not corrected as soon as possible. Therefore it consists permanently . Sometimes it gets in sync after long running, but not always.

saagarjha commented 4 months ago

Sorry, it seems like you posted that while I was replying! Your video seems like it is of the visionOS simulator. Can you reproduce it on actual hardware?

xplorer256 commented 4 months ago

Unfortunately no. Maybe there are some users with a real device here?

saagarjha commented 4 months ago

If you are on the simulator, you are almost certainly running into bugs in the Metal rendering path that I implemented as a stopgap there for testing. You can see the implementation of this here: https://github.com/saagarjha/Ensemble/blob/main/visionOS/ImageBufferView.swift. In the simulator the CoreAnimation path does not work at all, and software rendering is far too slow (set mask = true to try this if you want). The Metal renderer does but I haven't had the time to polish it so it has bugs like these, and I have it disabled on real hardware. I do at some point see switching entirely to Metal but that would be after I fix this and I just haven't gotten around to it.

xplorer256 commented 4 months ago

I do not quite understand, why this is exactly ONE SECOND frame rate. So I thought there must be anywhere something that reduces the frame rate to exactly 1 second. Could it be the TCP transport ? Or the video decoder?

saagarjha commented 4 months ago

https://developer.apple.com/documentation/quartzcore/cametallayer/2887086-allowsnextdrawabletimeout

xplorer256 commented 4 months ago

I will look somewhat closer at the ImageBufferView now...

xplorer256 commented 4 months ago

Made the test!!!

Did set : var mask: Bool = true static var useMetal: Bool = false

And what??? IT WORKS ! So there must be a bug in the Metal renderer. metalLayer.allowsNextDrawableTimeout does not solve the problem.

xplorer256 commented 4 months ago

.. another possible solution: prevent the redrawin g, if a resizing action is active.

saagarjha commented 3 months ago

I'm going to lump this into #15