search

marek-simonik / record3d_unity_demo

iPhone X -> Unity VFX Graph demo (real-time Point Cloud streaming)
https://record3d.app
129 stars 19 forks source link

stream video blink in 8/16 update. #8

Open idenn opened 2 years ago

idenn commented 2 years ago

Hi, I've tried the latest version, and it works great! However, the FaceID stream video seems to blink noticeably at a very high speed frequency. I think it's the limit of FaceID's record frequency in 30fps, but the scene is in 60fps or higher. Tried to limit the scene to 30fps but had no luck. Any solution? Thanks!

marek-simonik commented 2 years ago

I think this issue might be related to the spawn rate of VFX graph particles. I am going to take a closer look at this in the next few days and will notify you once I find the cause of the bug.

marek-simonik commented 2 years ago

I apologize for the delay. I was finally able to reproduce this issue, although only for the Record3D_Simple_Streaming object — did you experience flickering even with the Particles object?

After investigating the issue, I found that the flickering likely occurs due to the particles' Lifetime property being set to 1/FPS. This assumes that the iPhone/iPad will send new data exactly at FPS frame rate. But in practice, there can be delays and new camera data can arrive with slight delay, which what (I think) causes the occasional blinking, i.e. particles can sometimes die before new frame has been received. This is just a hypothesis, which might be wrong.

How to "fix" this?

To get rid of the flickering, I came up with a workaround specifically for the Record3D_Simple_Streaming object; all you need to do is to change the Record3D_Simple_Streaming VFX Graph so that it looks like the one in the attached image. Let me know if that worked for you.

Explanation

Since there are no visual effects and no dynamics in the Record3D_Simple_Streaming VFX Graph (only the raw stream is being shown), I unchecked the Lifetime checkbox in the Record3D_Simple_Streaming VFX Graph. This means that once the initial 307200 particles (for a 640x480 px frame) get spawned, none of them will die. Then, in the Update Particles section of the VFX Graph, I added a Set [Position|Color] from Map blocks (same as in Initialize Particle context). This ensures that the position and color of the "immortal" (already spawned) particles gets updated with the new camera data in each rendering loop.

I am not experienced with Unity / VFX Graph, but I feel like a cleaner solution (from the perspective of supporting continuous spawning of particles) would be to somehow use Events to trigger particle spawning and particle dying events. Maybe someone with more VFX Graph experience can help?

revised-vfx-graph