Memory bump after loading MTKView

[zhuhuihuihui]

I've tried my best to follow the best practice, but I'm still experiencing a big memory bump after loading MTKView using MetalPetal.

The image itself that I loaded from camera roll is about 1.9MB. When not using MTKView/MetalPetal, just load the photo into a standard 'UIImageView', the app uses 87MB in memory. As soon as I replace the 'UIImageView' with MTKView, the memory instantly bump to 1.59G and my phone gets really hot in like 3 seconds. Maybe I'm not using it correctly, I'll post my code blow.

Setup

fileprivate lazy var context: MTIContext? = {
        let options = MTIContextOptions()
        guard let device = MTLCreateSystemDefaultDevice(),
            let context = try? MTIContext(device: device, options: options) else { return nil }
        return context
    }()

    fileprivate lazy var renderView: MTKView = {
        let view = MTKView(frame: .zero, device: context?.device)
        view.delegate = self
        view.translatesAutoresizingMaskIntoConstraints = false
        view.layer.isOpaque = false
        view.autoresizingMask = [.flexibleWidth, .flexibleHeight]
        return view
    }()

    fileprivate lazy var renderRequest: MTIDrawableRenderingRequest = {
        let request = MTIDrawableRenderingRequest()
        request.drawableProvider = self.renderView
        request.resizingMode = MTIDrawableRenderingResizingMode.aspect
        return request
    }()

   inputImage = MTIImage(cgImage: cgImage, options: [MTKTextureLoader.Option.SRGB: false], isOpaque: false)

Rendering

public func draw(in view: MTKView) {
        brightnessFilter.inputImage = self.inputImage
        contrastFilter.inputImage = brightnessFilter.outputImage
        saturationFilter.inputImage = contrastFilter.outputImage
        guard let ajustedImage = saturationFilter.outputImage else { return }
        var outputImage = ajustedImage

        if let appliedFilter = appliedFilter {
            appliedFilter.inputImage = ajustedImage
            if let filteredImage = appliedFilter.outputImage {
                outputImage = filteredImage
            }
        }

        do {
            try autoreleasepool {
                try self.context?.render(outputImage, toDrawableWithRequest: renderRequest)
            }
        } catch {
            print(error)
        }
    }

I'm testing on iPhone XS. Please let me know if you need more info.

[YuAo]

What’s the pixel size of the image you are trying to process?

It seems you are doing lots of unnecessary redraw. MTKView typically draws at 60 frames per second. Are you trying to animate the image? If not, try using MTIImageView or set the isPaused property of MTKView to true.

You can also try context.reclaimResources() after each draw.

BTW, can you attach a sample project to demonstrate the issue? The image rendering demo of MetalPetal does not have the problem you mentioned.

[zhuhuihuihui]

Thanks for your response YuAo!

I'm using a standard photo taken by an iPhone, its pixel size is 4032 × 3024

Setting isPaused is a great idea. It helped me a lot on saving CPU usage, since now I can re-render only when user decide to change/adjust their filters. I can feel it's not burning my phone anymore. Thanks!

On the other side, I think context.reclaimResources() is not helping on saving memory. The initial render took about 500MB, and every time I re-render it'll bump up 500MB until it reaches around 1.6G.

I'm working on a demo right now, and I'll send it to you when it's ready.

BTW, I have another question.

Does your MTIColorLookupFilter take hald image as input, example:

if not, do you know if there's a way that I can convert this hald image into 2D Square CLUT in 512, the format that your MTIColorLookupFilter supports

Thanks

[YuAo]

MTIColorLookupFilter can handle color lookup images with the types defined in MTIColorLookupTableType. Ref: https://github.com/MetalPetal/MetalPetal/blob/master/Frameworks/MetalPetal/Filters/MTIColorLookupFilter.h

If you have any software that can use the HaldCLUT you provide, you can do the conversion: Just apply the HaldCLUT you provide to an identity 512x512 CLUT image.

[zhuhuihuihui]

Do you know any software that can handle this conversion? And also what is an identity 512x512 CLUT image? I tried to search for one, but I'm not confident on the searching result. Sorry I have very little experience on this field.

[YuAo]

Open source softwares like ImageMagick, GIMP, etc can handle HaldCLUT. Ref: https://rawpedia.rawtherapee.com/Film_Simulation

Here's one 512x512 CLUT: https://github.com/BradLarson/GPUImage/blob/master/framework/Resources/lookup.png

[zhuhuihuihui]

Hi @YuAo ,

Thanks for waiting. I've put up a demo to show you what I was talking about. Here's the link to the demo: https://github.com/zhuhuihuihui/MTIImageDemo

As you can see once you run this demo. The initial load took about 500MB, as soon as I adjust any filter values. It bumps up to 1.5G. I think this is something that belongs to the framework. Please share your thoughts.

Thanks, Scott

[YuAo]

Looks like your drawable is taking up too much memory. The frame size of your MTKView instance is 4032x3024, and the content scale factor is 3 on iPhone X. So one drawable will take 400MB+ memory. A MTKView will make 2-4 drawables for reuse so your memory bumps up to 1.5GB.

I don't think you need a view that big.

BTW, you can use MTIImageView to display a MTIImage. You can take a look in its source code to see how to use isPaused and how to properly set the contentScaleFactor.

[zhuhuihuihui]

Hi @YuAo

Thanks for the response. The reason why I need a view with that big frame size is because I'm using it in a scrollview so that the image itself is zoomable, and to archive that I have to frame the ImageView/MTKView with image's original size which is 4032x3024. Do you think MTIImageView could address this situation?

The thing that I don't understand is why would one drawable takes 400MB for a MTKView with big frame size, while UIImageView with same frame size doesn't take as much. It'd be great if you can share more thoughts on this. Meanwhile I'll also look into content scaleFactor.

Thanks, Scott

[YuAo]

width (4032) contentScaleFactor (3) height (3024) contentScaleFactor (3) bytesPerPixel, bgra (4) = 4032 3 3024 3 4 = 438,939,648 = 418.60546875 MB

An UIImageView uses a different approach, it just send what it needs to draw to the render server. You can also make use of this by making a CGImage from a MTIImage and display that image using an UIImageView.

Also, if you just replace your MTKView to MTIImageView and let the MTIImageView set the contentScaleFactor for you the memory usage should be much lower.

BTW. Even if you'd like to make the image zoomable you still do not have to use a 4032x3024 view. You can place a full screen view, handle the pinch and pan gesture to determine which part of the image is visible to the user and use a crop filter to crop/scale the image and display only that part of the image.