search

scier / MetalSplatter

Render Gaussian Splats using Metal on Apple platforms (iOS/iPhone/iPad, macOS, and visionOS)
MIT License
239 stars 27 forks source link

show ply model with passthrough video #4

Closed Yang-Xijie closed 6 months ago

Yang-Xijie commented 6 months ago

Hi, thanks for creating this repo!

I'm curious about showing ply model with passthrough video. Currently, the sample app in this repo and the app uploaded to App Store don't support passthrough video. With passthrough video, it will be safer to walk around in a scene model and it will be easier to put an object model on the desk.

However, as mentioned in https://developer.apple.com/documentation/visionos/creating-fully-immersive-experiences :

With a fully immersive experience, you’re responsible for everything that appears onscreen. The system hides passthrough video and displays the content you provide, showing the person’s hands only when they come into view. To achieve the best performance, use RealityKit or Metal to create and animate your content.

And, as mentioned in https://developer.apple.com/documentation/compositorservices/drawing_fully_immersive_content_using_metal :

Don’t include any style modifiers on a space that contains a CompositorLayer type. The system automatically configures a space with CompositorLayer content as fully immersive, and ignores any style modifiers.

I wonder if you have any idea on rendering both gaussian splatting model with Metal and passthrough video... My simple idea is to add stereo passthrough video as the background if the opacity doesn't reach 1.0 (pixel color = color accumulated + (1 - opacity) * passthrough video color). Do you think this is feasible?

Yang-Xijie commented 6 months ago

a post rather related: https://discussions.unity.com/t/show-passthrough-camera-feed-in-virtual-reality-fully-immersive-app/337732

It seems that visionOS doesn't support passthrough video when using CompositorLayer currently...

scier commented 6 months ago

I'd love to support rendering with passthrough video (i.e. not just in a full immersive space), but as you've found, that's impossible on visionOS 1.0 (or 1.1). The reason for this is explicit privacy restrictions: Apple doesn't want apps to be able to access the passthrough video. Moreover, it's driven by the basic technical architecture of visionOS created to enforce this policy: rendering normally takes place out-of-process (through RealityKit's purpose-built rendering engine), and passthrough video (or gaze data or other privacy-sensitive information) is never even sent into the app's process. However, Metal rendering needs to occur in-process.

It seems possible that someday Apple might send the results of the Metal render pass out of process for compositing. When that happens I'd love to implement this! In the meantime, we're stuck with the full immersive space, because RealityKit's renderer definitely isn't built to be flexible enough to render gaussian splats.

Yang-Xijie commented 6 months ago

Thanks for your reply!

scier commented 6 months ago

Oh, I'll also add -- I interpreted this issue as visionOS-specific, and so is my response. Rendering over passthrough on an iPhone via ARKit does not have these limitations, and should in theory be possible (may require tweaks to the library in theory, e.g. to the stored alpha values)