multiple 3d clips playing via timeline inquiry

[PockPocket]

Hi Marek,

I hope you are doing great.

I've been playing a lot with the project in the past year, making compositions out of multiple 3d clips.

Playing and working with multiple clips via the timeline works well, only thing is that there is a small fps drop (around 10fps) every time a new 3d clip is activated.

I was wondering if you'd knew of any way's of making successive 3dclips load in the scene without causing the jitter that comes with the fps drop ?

Sending good vibes your way, Mathieu

[marek-simonik]

Hi Mathieu,

I'm not sure what causes the FPS drop, but my first guess is that it might be related to the initialization of VFX Graph for each newly added/activated clip.

If displaying only the point cloud without any effects is sufficient for your purposes, then you might want to test whether decreasing the number of particles of the VFX Graph effect mitigates the FPS drop; here is a relevant guide that explains how to reduce the amount of particles: https://github.com/marek-simonik/record3d_unity_demo/issues/8#issuecomment-1234467544

In every case, I recommend to test if reducing the number of spawned particles of the VFX Graph has any effect on the FPS stability.

All the best, Marek

[PockPocket]

Hi Marek,

I hope you’re well!

I've been diving back into the project, having explored Depthkit and its plugin, which allowed me to play 3D videos captured with the Kinect Azure at a stable 30fps in VR, with VFX Graph at 5 million points. Although that workflow was promising, I found the recordings not as smooth and that filming with my iPhone offers better portability and practicality, as it doesn't require a PC connection during capture.

So, I'm back at it, still looking for a way to address the FPS limitations. Currently, my R3D clips max out at 21-22fps when played in play mode, through the timeline. I’ve tried reducing the point count, but unfortunately, that hasn’t improved the frame rate. Are you also experiencing this 21-22fps limit on your end?

Additionally, I haven’t yet figured out how to eliminate the lag when loading successive R3D clips from the timeline. There’s a noticeable FPS drop and lag each time a new R3D file is loaded into the scene. I’d appreciate any advice on how to maintain smoother transitions or improve FPS stability, especially for previously filmed R3D files imported into the asset folder.

Thanks a lot for your insights, and sending good vibes your way!

Best, Mathieu

[marek-simonik]

Hi Mathieu,

do you use the the simple Record3D_Simple_Streaming VFX graph or do you use the VFX graph effect created by Keijiro Takahashi (the Particles object)? If you use the simple graph, then it should be easier to get more performance.

Currently, I don't have access to a computer with Unity, so I cannot test by myself, but based on the second screenshot you shared, your VFX graph seems to be different than what I recommended in https://github.com/marek-simonik/record3d_unity_demo/issues/8#issuecomment-1234467544

In particular, your VFX graph sets lifetime for the particles in the "Initialize Particles" section (so that the particles repeatedly die and get respawned, which affects performance) and it also appears as if you are not updating the Position and Color attributes of the already spawned particles in the "Update Particles" section of the VFX graph. The number of spawned particles in the "Spawn" section is also very high; try to manually lower to e.g. 1 000 000 or to an even lower number.

Please try to replicate the VFX graph showed in https://github.com/marek-simonik/record3d_unity_demo/issues/8#issuecomment-1234467544 — in particular:

• Uncheck "Set Lifetime Random (Uniform)", so that particles don't dies (therefore the same particle's Position and Color attributes can be updated in each frame instead of creating a new stream of particles in each frame).
• Make sure that in the "Update Particle" section, you update the Position and Color attributes of each particle from the textures (as is done in the "Initialize Particle" section).
• See what happens when you lower the particle count in the "Spawn" section.

If there would still be performance issues, try lowering the number of spawned and/or initialized particles (in the "Spawn" and "Initialize Particle" sections) to absurdly low values (e.g. to 1 000 particles). If the performance would be as bad as before, then the bottleneck might be somewhere else.

[PockPocket]

I've replicated the graph and tried to put really low value to spawn and initialized particles (less than 1000) and the 20 fps bottleneck was still there. also the colors of the texture is now flashing red, blue, yellow repeatedly, which wasn't happening using my previous graph 🤔

any other idea on what could cause this bottleneck ?

I've tried different new projects with .r3d and vfx graph and it is alway's present in all of them.

thanks for your help, it is really appreciated!

[marek-simonik]

I still cannot try it in Unity myself, so let me at least ask a few questions:

1. What happens if you set just 1 spawn and initialize particle? Will you still get only 20 FPS?
2. How many clips do you have in the Timeline? Do you observe 20 FPS even with a single video in the Timeline?
3. Are you trying to replay LiDAR videos which were recorded with the "Higher-quality LiDAR recording" option enabled in the Settings section of Record3D? Such videos will be saved with 1920x1440 px RGB images. Perhaps that might be the bottleneck (i.e. slower loading and decoding of the high-resolution JPG images)?
4. Do you see 20 FPS even with 3D videos recorded via the selfie FaceID camera?

[PockPocket]

1. still 20 fps

2. one clip is active in the timeline

3. I was recording with "High-quality LiDAR recording" option enabled. I did a test with the option disabled and the playback was still 20 fps when in play mode.

4. 3d videos recorded with selfie FaceID camera plays at 70 fps but there's a bump every 3 seconds where the fps drops below 30 fps . this is also present when values for spawn and initialized is set to one in the graph you recommended me to use. out of curiosity I also tried it with my previous graph and the result was exactly the same with the fps and bumps. any idea on what is causing this bump ?

update 4. I brought this convo to chat gpt and it suggested me to verify if "Use Incremental GC" was enabled in the Player settings. enabling it kinda solved the bumps (made them smoother at least) ! I can play the clip with either one of the graphs, with 4M active points playing at 45-55 fps (it jumps from 45 to 55 every 1.5 sec aprox, but overall visualization is smooth) on my RTX2060 gaming laptop.

also if you have any other ideas on what could be the bottleneck issue for the LiDAR captures, i'm keen to try more debugging.

cheers,

Mathieu