Export Green Screen in Swift

[ahmedsafadii]

how it's possible to export Green Screen to asset or video, I use the example here Green Screen in Swift & PixelKit http://blog.hexagons.se/blog/green-screen-in-swift-pixelkit/

[heestand-xyz]

Hi, you can record a video with the RecordPIX

[ahmedsafadii]

I mean this is my code

let content = VideoPIX() content.load(fileNamed: "superman", withExtension: "mov")

let image = ImagePIX() image.image = UIImage(named: "city")

let key = ChromaKeyPIX() key.input = content key.keyColor = .green

let blend = BlendPIX() blend.blendingMode = .over blend.inputA = image blend.inputB = key

let final: PIX = blend final.view.frame = view.bounds view.addSubview(final.view)

how it's possible to render the video and save it with blending?

[heestand-xyz]

So PixelKit is a realtime based framework. It dose not have a timeline. Tho it's still possible to export a video! Depending on the quality you are looking for it can be done in different ways. All ways make use of the RecordPIX.

The quick and dirty way:

• Start recording: recordPix.startRec(name: "video-name")
• Start the video: content.play()
• Wait with a timer. Get the duration in seconds: content.duration
• Stop the recoding: recordPix.stopRec({ url in })

Let me know if you want a more frame accurate method.

[ahmedsafadii]

Wait with a timer. Get the duration in seconds: content.duration

you mean that need to check video duration with the playing status so I decied to stop recording

my code

        let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)
        do {
            try? record.startRec(name: "testMe")
            content.play()
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        } catch {
            print("Not Working")
        }

[ahmedsafadii]

Also, I got an error

PixelKit #413 ERROR [2] RecordPIX "record" >>> Rec Stop A. Writer or Input not found. Error Record render("Rec Stop A. Writer or Input not found.") my code

        let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)

        try? record.startRec(name: "testMe")
        content.play()
        content.listenToDone {
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        }

[heestand-xyz]

Don't forget to connect the final to the record:

record.input = final

Then setup the timer something like this:

do {
    try record.startRec(name: "testMe")
    content.play()
    RunLoop.current.add(Timer(timeInterval: content.duration!, repeats: false, block: { _ in
        record.stopRec({ url in
            print("Finish", url)
        }, didError: { error in
            print("Error Record", error)
        })
    }), forMode: .common)
} catch {
    print("Not Working")
}

[heestand-xyz]

The listenToDone way should work too. Forgot I added that method. Nice find! Let me know if it works!

[ahmedsafadii]

Thank you for helping but I got the same error even after doing the input thing

PixelKit #413 ERROR [2] RecordPIX "record" >>> Rec Stop A. Writer or Input not found. Error Record render("Rec Stop A. Writer or Input not found.")

Code

let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)

        record.input = blend

        try? record.startRec(name: "testMe")
        content.play()
        content.listenToDone {
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        }

[heestand-xyz]

Ok, so first make sure startRec(name:) worked by either try! or a do catch block. Then double check listenToDone is not being called to early. That might cause the crash. If the listenToDone method dose not work (if so I'm sorry), then try the timer method.

[heestand-xyz]

maybe try to start the record and playback in the video load callback, like this:

content.load(fileNamed: "bo", withExtension: "mp4", done: { resolution in
    try! record.startRec(name: "testMe")
    content.play()
    content.listenToDone {
        record.stopRec({ url in
            print("Finish", url)
        }, didError: { error in
            print("Error Record", error)
        })
    }
})

[ahmedsafadii]

it export but the problem now with the video it's very strange

writer Optional(<AVAssetWriter: 0x600002130770, outputURL = file:///Users/ahmedsafadi/Library/Developer/CoreSimulator/Devices/837DB0F3-5156-4254-8E54-F94029FB9B3E/data/Containers/Data/Application/CBB87496-B419-472E-98B2-775E8FCD208C/Documents/pixelKit/renders/8624EE74-1295-463C-9D69-0910E38F3DCD/testMe.mov, outputFileType = com.apple.quicktime-movie>) PixelKit #315 ERROR [2] RecordPIX "record" >>> Exported frame failed, writer status: 1. PixelKit #323 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)" Finish file:///Users/ahmedsafadi/Library/Developer/CoreSimulator/Devices/837DB0F3-5156-4254-8E54-F94029FB9B3E/data/Containers/Data/Application/CBB87496-B419-472E-98B2-775E8FCD208C/Documents/pixelKit/renders/8624EE74-1295-463C-9D69-0910E38F3DCD/testMe.mov

Preview looks like

Output looks like

so do you know what the problem could be?

I just want to say that this is amazing efforts from this library

[ahmedsafadii]

I keep getting this error when the recording is start

ixelKit #367 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #370 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #372 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #376 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #380 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #384 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #387 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #392 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #396 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #402 WARNING [1] VideoPIX "video" Resource >>> Texture genr

[heestand-xyz]

Wow that dose not look good. So I think you can disregard some of the errors (-6660). I'll check them. So print double check the resolution of your blend pix with .renderResolution. The resolution is always derived from inputA. You can swap inputA and inputB and use .under as the blending mode. Thanks for the kudos.

[heestand-xyz]

Let me know if swapping the blend inputs help.

blend.blendMode = .under
blend.inputA = key
blend.inputB = image

[heestand-xyz]

If you need to flip the video (some known bugs with vertical flipping are not fixed yet), use ._flipY()

[ahmedsafadii]

@hexagons I did the swap and I guess it's work but the frame per second looks too slow

[ahmedsafadii]

also, I guess the green background for the video didn't go

[heestand-xyz]

try blend.placement = .aspectFill

[ahmedsafadii]

@hexagons yes the green hide but the record looks like the fps is soo slow after export the video is very slow show up it like 10 frame per second

[heestand-xyz]

ah ok, try recordPix.timeSync = false

[heestand-xyz]

or recordPix.realtime = true

[ahmedsafadii]

recordPix.timeSync = false ( make the video too fast) recordPix.realtime = true ( make the video slow like it was )

[heestand-xyz]

So this might be hard to fix with this method. PixelKit is a realtime software, so if your device can't playback in realtime it can't record in realtime. Tho there is a manual override mode. It's a bit more complex to setup. This will render frame accurate video. Let me know if you want try learn the method and I can write a post about it.

[heestand-xyz]

It involves turning off realtime mode in the engine, seeking the video to each frame and rendering manually by saving the .renderedImage of the PIX.

PixelKit.main.render.engine.renderMode = .manual
PixelKit.main.render.engine.manuallyRender { print("frame rendered") }

[ahmedsafadii]

First, thank you very much for this kind of fast and helping

what I discover that I guess the export setting need to be controlled by the user since you export .mov with a very good setting for good quality, that what makes it slow my guessing, I try on a real device it was faster but not smooth, I recommend creating documentation for export since this will make this library very famous.

and again thank you

Output test.mov.zip

[heestand-xyz]

I will work on writing more documentation. I will also look at adding a quality parameter. Thank you!

[ahmedsafadii]

I will be watching it more and more, keep this amazing effort, and I will look to see it In future <3