scier
commented
4 months ago What you're seeing -- which I've dubbed WobbleSplat, trademark pending -- is a combination of two things. First, the Vision Pro extrapolates missed frames by reprojecting previous frames, using those frames' depth to warp the textures. VP needs to render at 90fps, so if the splat is rendering at anything less than that, it'll miss a frame which is super-ugly and disorienting in VR, so the VP creates a new one to match the user's latest pose. Reprojection isn't perfect -- especially around depth discontinuities, i.e. edges of foreground objects -- but it depends on having good depth to work at all.
Which brings us to the second issue: rendering depth is hard with splats (because of the transparency and layers involved). For instance, if there's a large, nearly-invisible splat, the depth buffer will still show that "ghost bubble" as the closest object for those pixels, and so the reprojection will happily warp the pixels as though they're closer to you than they actually are, and you'll see a warped bubble.
The faster you move, the bigger (i.e. slower to render) the splat model, and the more noisy large translucent splats there are in the model, the more this will all stand out.
If you want to see what things look like with no reprojection at all -- well, that's hard, it's automatic. But if you want to turn it off "halfway", you can turn off depth: in SplatRenderer, set depthStateDescriptor.isDepthWriteEnabled = false. Then the depth buffer will contain entirely 1.0. The reprojection will then treat everything as though it's really far away. Which means it does just fine when you just turn your head in place, but if you move side-to-side, it can't compensate, so foreground objects jitter a lot and seem to "lag" (since they can't be moved to the "right" place).
There are a few ways to fix this, none easy (sorry):
- Render at 90fps. If you can improve the efficiency of the renderer enough to do that, I'll be quite happy. You'll notice the WobbleSplat effect is more pronounced on bigger splats; this is why.
- Have very clean models with no translucent "floaters", or smaller models with fewer/smaller splats. This helps a lot, but it's perfect; you'll still see wobbles around depth discontinuities (foreground edges) and truly transparent areas.
- Write better depth values; e.g. write depth as an alpha-blended composite, similar to colors. This may be feasible, but so far Metal has defeated every attempt I've made to do it efficiently. Again, I'd love to see a patch that did this (well)! Although it still wouldn't be perfect (see the previous bullet)
JiamingSuen
jerbeers
I noticed this strange wobbling artifact (especially on the depth edges) on Vision Pro. The same issue can be reproduced with a smaller splat file (34K points) and the closed-sourced version of the app on the App Store. I'm on visionOS v1.1.2.
https://github.com/scier/MetalSplatter/assets/8616975/16529d0e-abd7-4267-8964-98234464f43e
https://github.com/scier/MetalSplatter/assets/8616975/d5532caa-a25b-48c8-b540-b7241a3867fd
Hope this can be easily fixed. Thanks for this amazing repo!