Request for ability to record videos with transparency

[geoffsdavis]

Hi, thanks so much for making this tool! I have a feature request:

I'm trying to create an application that can export SceneKit views as .mov files, as a kind of authoring tool for videos of animations. However, I would like the exported .mov files to have transparent backgrounds. Would it be easy to add the ability to record SCNViews (and other views) with transparency preserved? Currently, SCNRecorder exported .mov files always show transparent pixels as black. I cloned your repo and tried to implement transparency myself, but unfortunately my level of expertise with pixel formatting, codecs, etc is not sufficient. Perhaps it's easy if you know what you're doing, though, since it might just be a matter of proper settings.

If it's any help, the GPU debugger reported the CAMetalLayer Display Drawable in my SCNView as using bgra8Unorm_srgb, and when I examined individual pixels in the GPU debugger, alpha was 0 where the pixels were transparent. My app correctly displays the SCNView with the transparent background, but the exported .mov file does not seem to have it.

Here's how I manually create a .mov with a transparent background. When I export a video clip with a transparent background from Adobe Premiere, I choose Quicktime (.mov) and set the code to "Apple ProRes 4444 with alpha". Then, in the Mac Finder, I right click on the exported .mov file and select "Encode Selected Video Files" from the menu. Then I choose "HEVC 1080p" from Setting, and click the "Preserve Transparency" checkbox. After a bit, the Mac produces a new .mov file, which I can then bring into Xcode and play in any application. The .mov's transparent background will display correctly in the new application.

However, when I produce a .mov file from SCNRecorder, then attempt to use the Finder's encoding tool with the above steps, Finder gives me an error if I check the "Preserve Transparency" checkbox:

/////// avconvert: invalid configuration (preset name PresetHEVC1920x1080WithAlpha) Presets compatible with file: Preset640x480 Preset960x540 Preset1280x720 Preset1920x1080 Preset3840x2160 PresetAppleM4V480pSD PresetAppleM4V720pHD PresetAppleM4V1080pHD PresetAppleM4VAppleTV PresetAppleM4VCellular PresetAppleM4ViPod PresetAppleM4VWiFi PresetAppleProRes422LPCM PresetAppleProRes4444LPCM PresetHEVC1920x1080 PresetHEVC3840x2160 PresetHEVCHighestQuality PresetHighestQuality PresetLowQuality PresetMediumQuality ///////

Thanks again, and no worry if this feature is too much bother to build.

Yours, Geoff

[v-grigoriev]

Hi @geoffsdavis, When I was deciding about supporting alpha or not, the main reason not to support was that I didn't find a way to configure SCNView (especially ARSCNView) to support pixel formats with alpha channel. In ARKit it doesn't make a lot of sense and probably I forgot to try with SCNView.

Of course, it can easily be achieved with MTKView or UIView backed with CAMetalLayer, but the main purpose of the library is to record SCNView/ARSCNView.

May I ask you to share your configuration of SCNView, so I can do some experiments?

Also, there are a few more things to be resolved:

• PixelBufferProduce should become format dependant, since as far as I know only AVVideoCodecType.hevcWithAlpha can support pixels with alpha. (AVVideoCodecType.proRes4444 should work too, but I never worked with it)
• If the pixels have alpha, but the format does not support it, how should the lib deal with it? Ignore and show as black? Or merge with a predefined configurable background?

They are just a matter of choice. I don't see any reason why this might not work. I don't commit myself to any time but will take a look at this with priority.

Vlad

[v-grigoriev]

@geoffsdavis Done. Try the 2.6.0 version. To start recording videos with transparent background use at the minimum: try sceneView.startVideoRecording(videoSettings: VideoSettings(codec: .hevcWithAlpha()))