Video completely distorted and glitchy

[LeonardRockstar]

When using SceneKitVideoRecorder on an ARSCNView, the video is completely distorted. This is happening on my iPhone 7 Plus running Beta 10.

[okaris]

Wow. Strange yet cool. Will update iOS now to see whats wrong.

[okaris]

Tested on iPhone SE iOS 11 Beta 10. Can’t reproduce.

[LeonardRockstar]

Weird, I've even created a new AR Kit project and I still see the error. What device were you using?

[okaris]

I was on iPhone SE Can you share your setup options and pod version?

[LeonardRockstar]

I was testing on an iPhone 7 Plus with SceneKitRecorder 1.0.3. I created a new Xcode Project with the ARKit template.

[LeonardRockstar]

I've just tested the same project on my iPad Pro. The video is not distorted, but it is really choppy.

[okaris]

Can you test this issue with 1.2.0 please?

[okaris]

As you showed there is a bug in 1.2.0 Can you try this with 1.1.0 until i fix 1.2.0 please

[LeonardRockstar]

The issue still persists with 1.1.0

[okaris]

Please try with sample configuration. Don't forget to add prepare() on viewDidAppear

[LeonardRockstar]

I've tried the sample configuration, but the video still looks like that.

[mariedm]

I have the same issue. I'm using an iPhone 6S on iOS 11, I think I was on the 1st beta. I'm currently updating to the last one (beta 10). I'll retry after and tell you.

[mariedm]

I updated my iPhone 6S to iOS 11 beta 10. I retested and I'm still having the same issue.

[okaris]

I have a strange theory and a really bad way to test it.

@LeonardRockstar @mariedm Can you paste the following code on line 21 in PixelBufferFactory.swift just under let destinationTexture = currentDrawable.texture

switch destinationTexture.pixelFormat {
        case .invalid: print("invalid"); break;
        case .a8Unorm: print("a8Unorm"); break;
        case .r8Unorm: print("r8Unorm"); break;
        case .r8Unorm_srgb: print("r8Unorm_srgb"); break;
        case .r8Snorm: print("r8Snorm"); break;
        case .r8Uint: print("r8Uint"); break;
        case .r8Sint: print("r8Sint"); break;
        case .r16Snorm: print("r16Snorm"); break;
        case .r16Uint: print("r16Uint"); break;
        case .r16Sint: print("r16Sint"); break;
        case .r16Float: print("r16Float"); break;
        case .rg8Unorm: print("rg8Unorm"); break;
        case .rg8Snorm: print("rg8Snorm"); break;
        case .rg8Uint: print("rg8Uint"); break;
        case .rg8Sint: print("rg8Sint"); break;
        case .a1bgr5Unorm: print("a1bgr5Unorm"); break;
        case .abgr4Unorm: print("abgr4Unorm"); break;
        case .bgr5A1Unorm: print("bgr5A1Unorm"); break;
        case .r32Sint: print("r32Sint"); break;
        case .r32Float: print("r32Float"); break;
        case .rg16Unorm: print("rg16Unorm"); break;
        case .rg16Snorm: print("rg16Snorm"); break;
        case .rg16Uint: print("rg16Uint"); break;
        case .rg16Sint: print("rg16Sint"); break;
        case .rg16Float: print("rg16Float"); break;
        case .rgba8Unorm: print("rgba8Unorm"); break;
        case .rgba8Unorm_srgb: print("rgba8Unorm_srgb"); break;
        case .rgba8Snorm: print("rgba8Snorm"); break;
        case .rgba8Uint: print("rgba8Uint"); break;
        case .rgba8Sint: print("rgba8Sint"); break;
        case .bgra8Unorm: print("bgra8Unorm"); break;
        case .bgra8Unorm_srgb: print("bgra8Unorm_srgb"); break;
        case .rgb10a2Unorm: print("rgb10a2Unorm"); break;
        case .rgb10a2Uint: print("rgb10a2Uint"); break;
        case .rg11b10Float: print("rg11b10Float"); break;
        case .rgb9e5Float: print("rgb9e5Float"); break;
        case .bgr10a2Unorm: print("bgr10a2Unorm"); break;
        case .bgr10_xr: print("bgr10_xr"); break;
        case .bgr10_xr_srgb: print("bgr10_xr_srgb"); break;
        case .rg32Sint: print("rg32Sint"); break;
        case .rg32Float: print("rg32Float"); break;
        case .rgba16Unorm: print("rgba16Unorm"); break;
        case .rgba16Snorm: print("rgba16Snorm"); break;
        case .rgba16Uint: print("rgba16Uint"); break;
        case .rgba16Sint: print("rgba16Sint"); break;
        case .rgba16Float: print("rgba16Float"); break;
        case .BGRA10_XR: print("BGRA10_XR"); break;
        case .bgra10_XR_sRGB: print("bgra10_XR_sRGB"); break;
        case .rgba32Sint: print("rgba32Sint"); break;
        case .rgba32Float: print("rgba32Float"); break;
        case .pvrtc_rgb_2bpp_srgb: print("pvrtc_rgb_2bpp_srgb"); break;
        case .pvrtc_rgb_4bpp: print("pvrtc_rgb_4bpp"); break;
        case .pvrtc_rgb_4bpp_srgb: print("pvrtc_rgb_4bpp_srgb"); break;
        case .pvrtc_rgba_2bpp: print("pvrtc_rgba_2bpp"); break;
        case .pvrtc_rgba_2bpp_srgb: print("pvrtc_rgba_2bpp_srgb"); break;
        case .pvrtc_rgba_4bpp: print("pvrtc_rgba_4bpp"); break;
        case .pvrtc_rgba_4bpp_srgb: print("pvrtc_rgba_4bpp_srgb"); break;
        case .eac_r11Snorm: print("eac_r11Snorm"); break;
        case .eac_rg11Unorm: print("eac_rg11Unorm"); break;
        case .eac_rg11Snorm: print("eac_rg11Snorm"); break;
        case .eac_rgba8: print("eac_rgba8"); break;
        case .eac_rgba8_srgb: print("eac_rgba8_srgb"); break;
        case .etc2_rgb8: print("etc2_rgb8"); break;
        case .etc2_rgb8_srgb: print("etc2_rgb8_srgb"); break;
        case .etc2_rgb8a1: print("etc2_rgb8a1"); break;
        case .etc2_rgb8a1_srgb: print("etc2_rgb8a1_srgb"); break;
        case .astc_5x4_srgb: print("astc_5x4_srgb"); break;
        case .astc_5x5_srgb: print("astc_5x5_srgb"); break;
        case .astc_6x5_srgb: print("astc_6x5_srgb"); break;
        case .astc_6x6_srgb: print("astc_6x6_srgb"); break;
        case .astc_8x5_srgb: print("astc_8x5_srgb"); break;
        case .astc_8x6_srgb: print("astc_8x6_srgb"); break;
        case .astc_8x8_srgb: print("astc_8x8_srgb"); break;
        case .astc_10x5_srgb: print("astc_10x5_srgb"); break;
        case .astc_10x6_srgb: print("astc_10x6_srgb"); break;
        case .astc_10x8_srgb: print("astc_10x8_srgb"); break;
        case .astc_10x10_srgb: print("astc_10x10_srgb"); break;
        case .astc_12x10_srgb: print("astc_12x10_srgb"); break;
        case .astc_12x12_srgb: print("astc_12x12_srgb"); break;
        case .astc_4x4_ldr: print("astc_4x4_ldr"); break;
        case .astc_5x4_ldr: print("astc_5x4_ldr"); break;
        case .astc_5x5_ldr: print("astc_5x5_ldr"); break;
        case .astc_6x5_ldr: print("astc_6x5_ldr"); break;
        case .astc_6x6_ldr: print("astc_6x6_ldr"); break;
        case .astc_8x5_ldr: print("astc_8x5_ldr"); break;
        case .astc_8x6_ldr: print("astc_8x6_ldr"); break;
        case .astc_8x8_ldr: print("astc_8x8_ldr"); break;
        case .astc_10x5_ldr: print("astc_10x5_ldr"); break;
        case .astc_10x6_ldr: print("astc_10x6_ldr"); break;
        case .astc_10x8_ldr: print("astc_10x8_ldr"); break;
        case .astc_10x10_ldr: print("astc_10x10_ldr"); break;
        case .astc_12x10_ldr: print("astc_12x10_ldr"); break;
        case .astc_12x12_ldr: print("astc_12x12_ldr"); break;
        case .r16Unorm: print("r16Unorm"); break;
        case .rg8Unorm_srgb: print("rg8Unorm_srgb"); break;
        case .b5g6r5Unorm: print("b5g6r5Unorm"); break;
        case .r32Uint: print("r32Uint"); break;
        case .rg32Uint: print("rg32Uint"); break;
        case .rgba32Uint: print("rgba32Uint"); break;
        case .bc1_rgba: print("bc1_rgba"); break;
        case .bc1_rgba_srgb: print("bc1_rgba_srgb"); break;
        case .bc2_rgba: print("bc2_rgba"); break;
        case .bc2_rgba_srgb: print("bc2_rgba_srgb"); break;
        case .bc3_rgba: print("bc3_rgba"); break;
        case .bc3_rgba_srgb: print("bc3_rgba_srgb"); break;
        case .bc4_rUnorm: print("bc4_rUnorm"); break;
        case .bc4_rSnorm: print("bc4_rSnorm"); break;
        case .bc5_rgUnorm: print("bc5_rgUnorm"); break;
        case .bc5_rgSnorm: print("bc5_rgSnorm"); break;
        case .bc6H_rgbFloat: print("bc6H_rgbFloat"); break;
        case .bc6H_rgbuFloat: print("bc6H_rgbuFloat"); break;
        case .bc7_rgbaUnorm: print("bc7_rgbaUnorm"); break;
        case .bc7_rgbaUnorm_srgb: print("bc7_rgbaUnorm_srgb"); break;
        case .pvrtc_rgb_2bpp: print("pvrtc_rgb_2bpp"); break;
        case .eac_r11Unorm: print("eac_r11Unorm"); break;
        case .astc_4x4_srgb: print("astc_4x4_srgb"); break;
        case .gbgr422: print("gbgr422"); break;
        case .bgrg422: print("bgrg422"); break;
        case .depth16Unorm: print("depth16Unorm"); break;
        case .depth32Float: print("depth32Float"); break;
        case .stencil8: print("stencil8"); break;
        case .depth24Unorm_stencil8: print("depth24Unorm_stencil8"); break;
        case .depth32Float_stencil8: print("depth32Float_stencil8"); break;
        case .x32_stencil8: print("x32_stencil8"); break;
        case .x24_stencil8: print("x24_stencil8"); break;
      }

This huge switch is because printing the enum or its rawValue didn't work. I am getting bgra8Unorm_srgb and my theory is that somehow yours is different and causing this psychedelic recording.

[LeonardRockstar]

Just tried it, it keeps printing "invalid".

[okaris]

Interesting. Are you getting a whole video like this, or do you crash or do you have a video with one frame frozen?

[LeonardRockstar]

I'm getting the whole video like that.

[okaris]

Another idea: can you try with 30fps video please. This is really hard to debug when i cant reproduce the issue

[okaris]

@mariedm are you using 6s or 6s plus?

[LeonardRockstar]

Still looks the same when using 30fps.

[okaris]

@LeonardRockstar can you put the switch to SceneKitVideoRecorder.swift under line 74 ...frameBufferOnly = false

and also changing switch to

switch metalLayer.pixelFormat {

[LeonardRockstar]

It prints bgra8Unorm_srgb

[okaris]

That looks correct. One more thing to try. Can you pass in a block to updateFrameHandler of SceneKitVideoRecorder and get an image from there to see if its the same with the video.

[LeonardRockstar]

I don't know if I understand you correctly. The UIImage returned by PixelBufferFactory.make (and passed into updateFrameHandler) is also distorted, but looks a lot brighter than the video.

[GoodbyeCain]

same issue

[okaris]

@GoodbyeCain can you please share your device, os version, pod version so we can check if a pattern is forming.

[mariedm]

I’m using a regular 6S. In PixelBufferFactory: the switch statement on metalLayer.pixelFormat prints invalid. In SceneKitVideoRecorder: the switch statement on metalLayer.pixelFormat after framebufferOnly (line 64 for me) also prints invalid.

[mariedm]

If I put a breakpoint on updateFrameHandler in SceneKitVideoRecorder to get the image, here's what I get (much more white, or brighter like @LeonardRockstar said):

[okaris]

After a lot of googling and presumptions I think there is something wrong with the colorSpace of the buffer provided. https://stackoverflow.com/questions/34913005/color-space-mapping-ycbcr-to-rgb the example here looks much like this problem. Will try to manually convert ycbcr to rgb if colorSpace is invalid

[sanboxapps]

Love this repo, I am also getting the same issue on an iPhone 7+ running iOS 11 GM (15A372)

[okaris]

@sanboxapps thanks. Can you also try the script above please?

[sanboxapps]

I just tried the code on an iPad pro (10", iOS 11 GM), and got the same result as far as the switch statements I am getting invalid for ~both~ the factory method and bgra8Unorm_srgb for the recorder method I have tried with 30, and 60 fps it I am getting the same effect but with a slight difference in the colors from 30 to 60

[okaris]

Thanks alot for the input.

[sanboxapps]

No problem 👍 please let me know if you need further information, I would be happy to help.

[okaris]

Can you guys try this after replacing line 46 on Options.swift

kCVPixelBufferPixelFormatTypeKey as String: NSNumber(value: kCVPixelFormatType_32BGRA),

with

kCVPixelBufferPixelFormatTypeKey as String: NSNumber(value: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),

[sanboxapps]

got this exception: '_validateGetBytes:95: failed assertion rowBytes(1924) must be >= (2160).' on this line destinationTexture.getBytes(tempBuffer!, bytesPerRow: Int(bytesPerRow), from: region, mipmapLevel: 0) in PixelBufferFactory

[okaris]

Hmm. Can you try adding print(bytesPerRow, currentDrawable.layer.drawableSize.width, currentDrawable.layer.drawableSize) to PixelBufferFactory.swift under line #28 let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer) and one of the outputs please

[LeonardRockstar]

I'm getting the same error as @sanboxapps. The output is: 1932 1080.0 (1080.0, 1867.0)

[mariedm]

I updated the pod, when calling startWriting() I got Error Domain=SceneKitVideoRecorder.SceneKitVideoRecorder.PreparationError Code=1 "(null)".

I readded both switch statements. The one in SceneKitVideoRecorder is now printing bgra8Unorm_srgb(instead of invalid before).

Then in the Options file I changed kCVPixelFormatType_32BGRA to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and still got the same error, I can't record.

[okaris]

@mariedm if you are seeing bgra8Unorm_srgb as pixelFormat I don't know why but it should be fixed for you. Can you run the demo project in the latest pod please.

[mariedm]

I didn't manage to run the demo project but I recreated it. I could record and I have the audio (!) but the video is distorded. The switch statement in SceneKitVideoRecorder prints bgra8Unorm_srgb and the one in PixelBufferFactory prints invalid.

If I change to 30 fps I get the same outputs.

If I change to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange in Options I get the same outputs + _validateGetBytes:95: failed assertion rowBytes(1156) must be >= (1500) in PixelBufferFactory when it crashed on the line destinationTexture.getBytes(tempBuffer!, bytesPerRow: Int(bytesPerRow), from: region, mipmapLevel: 0).

[okaris]

Please try this when the video is colorful like above and share the output. Thanks all!

print(bytesPerRow, currentDrawable.layer.drawableSize.width, currentDrawable.layer.drawableSize)to PixelBufferFactory.swift under line #28 let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)

[mariedm]

This is the output: 3008 750.0 (750.0, 1294.0).

[okaris]

@warrenm I found your answer about Metal Video Recording on StackOverflow. This issue looks like a corruption you wrote on SO. Can you help us solve it?

[ghost]

Sure. The texture whose bytes you're copying probably has a format of MTLPixelFormatRGB10A8_2P_XR10_sRGB or something. As I said very explicitly in my answer on SO, that snippet depends on the source format being MTLPixelFormatBGRA8Unorm. You'll either need to figure out how to set the pixel format of the CAMetalLayer underlying the Scene Kit view to have that format, or else run a conversion step (fragment shader, compute shader, vImage, etc.) that transforms the pixel data into the necessary format. Naively copying from the drawable texture into a pixel buffer isn't going to cut it in this case.

[okaris]

@warrenm Thank you very much for your guidance. It provided some perspective to try other solutions.

@mariedm @LeonardRockstar @sanboxapps @GoodbyeCain Can you please run the latest (1.2.7) version of this project and let me know about the results please?

[LeonardRockstar]

It crashed at line 20 in PixelBufferFactory: var destinationTexture = currentDrawable.texture.makeTextureView(pixelFormat: .bgra8Unorm)

with the error: validateArgumentsForTextureViewOnDevice, line 1037: error 'source texture pixelFormat (MTLPixelFormatRGB10A8_2P_XR10_sRGB) not compatible with texture view pixelFormat (MTLPixelFormatBGRA8Unorm).' validateArgumentsForTextureViewOnDevice:1037: failed assertionsource texture pixelFormat (MTLPixelFormatRGB10A8_2P_XR10_sRGB) not compatible with texture view pixelFormat (MTLPixelFormatBGRA8Unorm).'`

[okaris]

@warrenm Can you suggest a source for how to convert these pixelFormats to one another? Especially I can not seem to find any information or documentation online for RGB10A8_2P_XR10_sRGB pixelFormat. The closest one I got was bgra10_XR_sRGB which is not probably close enough.

Also I don't believe that this pixelFormat is something should be returned by a MTLTexture coming from a SceneKit scene CAMetalLayer. Documentation tells the only following should be returned:

.bgra8Unorm .bgra8Unorm_srgb .rgba16Float

with bgra8Unorm_srgb being the default.

[mariedm]

I have the same crash & error outputs than @LeonardRockstar.

[ghost]

@okaris It's unfortunate that this breaking change occurred, and that neither the change nor the format itself is documented. I don't know if XR biplanar format can be read or sampled by a shader, but if it can, you could bind the framebuffer as a source texture and "re-render" the frame to a bgra8Unorm target (with a full-screen quad pass) in order to get around this limitation. That will measurably impact the performance of the recorder. You might be able to recover part of the framerate hit by not taking a copy and building a UIImage every frame; that's an extremely expensive operation that doesn't seem to be necessary for the primary function of the framework.