4S Crashes on Photo Capture etc : Memory leak?

[iamcam]

I have an iPod 4gen on iOS6 that doesn't exhibit this problem, so it's been frustrating to see my 4S/iOS5 crash with 100% reproducibility when I capture and recapture a photo with the still camera.

I basically show a live preview in one view, and have a another view that displays the captured photo - both use the same filter chain as the source. I run into problems when I tell the stillCamera to capture from the filter output, then set a temporary selectedImage as that new source image data. With the UIImage, I then reset all the targets in the chain so I can switch between filters on the still image. The camera preview is stopped ( [self.stillCamera stopCameraCapture]). Going back to live preview is simply a process of repeating the steps use in init to get fresh objects - sources and filters.

What appears to be happening is that there's some kind of memory leak going on, as each time I call [self.stillCamera capturePhotoAsJPEGProcessedUpToFilter... seems to increase the app's memory by a consistent amount. - depends on the device and the image size. The iPod will last longer, as its memory doesn't grow quite as fast. Sometimes the 4S dies nearly immediately upon taking a photo. Even with some optimizations I can get the memory usage down, but after a few shots it still crashes. I just can't figure out why the process of stopping the camera, saving, and restarting would be so expensive that the app would crash like that, much less how to prevent it. No errors or warnings, either.

Has anyone experienced something like this, or know a work-around? It's driving me nuts, and is frankly unacceptable.

Ugh.

[manjeettbz]

Hey ,

i am also working on the same issue. let me know if you find anything. thanks

[manjeettbz]

seems like its memory issue: in GPUImageStillCamera.h

---

if (!CGSizeEqualToSize(sizeOfPhoto, scaledImageSizeToFitOnGPU)) { CMSampleBufferRef sampleBuffer; GPUImageCreateResizedSampleBuffer(cameraFrame, scaledImageSizeToFitOnGPU, &sampleBuffer);

        dispatch_semaphore_signal(frameRenderingSemaphore);
        [self captureOutput:photoOutput didOutputSampleBuffer:sampleBuffer fromConnection:[[photoOutput connections] objectAtIndex:0]];
        dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);
        CFRelease(sampleBuffer);
    }
    else
    {
        dispatch_semaphore_signal(frameRenderingSemaphore);
        [self captureOutput:photoOutput didOutputSampleBuffer:imageSampleBuffer fromConnection:[[photoOutput connections] objectAtIndex:0]];
        dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);
    }

---

[iamcam]

Definitely seems that way. It's seems to be happening around the point of actually saving the photo. It's worse when I'm actually saving photos. My suspicion is that the app is requesting a large chunk of memory all at the same time, so between capturing the full-sized image and saving, there may be two copies of the same (large) data going through the system: one in GPUImage, and the other in the UIImage->disk.

Something interesting: one thing I'm seeing is that if I set a breakpoint before

[self.stillCamera capturePhotoAsImageProcessedUpToFilter:self.cropFilter 
withCompletionHandler:^(UIImage *processedImage, NSError *error) {

the debugger pauses in three places: once before the stillCamera capture, and three times immediately before the block executes. If I slowly step through those four steps, it seems the app behaves much better, but if I remove that breakpoint, all bets are off. Between breaks 1&1.1, is the asynchronous capture block in GPUImageStillCamera

capturePhotoAsImageProcessedUpToFilter:withCompletionHandler:block;

I'm not sure what happens between the other breakpoints yet. I'd say that probably every time the crash occurs, it's immediately after `[self captureOutput:photoOutput didSampleBuffer...] method, and before dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);

I've set memory reports throughout the method, and they would read like this (corresponding code below): Middle 1 - {mem} Middle 3 - {mem} Middle 3.2 - {mem} {crashes here if it's going to crash} After filter processing - {mem}

- (void)capturePhotoAsImageProcessedUpToFilter:(GPUImageOutput<GPUImageInput> *)finalFilterInChain withCompletionHandler:(void (^)(UIImage *processedImage, NSError *error))block;
{
    dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);

    [photoOutput captureStillImageAsynchronouslyFromConnection:[[photoOutput connections] objectAtIndex:0] completionHandler:^(CMSampleBufferRef imageSampleBuffer, NSError *error) {
        report_memory(@"Before filter processing");

        // For now, resize photos to fix within the max texture size of the GPU
        CVImageBufferRef cameraFrame = CMSampleBufferGetImageBuffer(imageSampleBuffer);

        CGSize sizeOfPhoto = CGSizeMake(CVPixelBufferGetWidth(cameraFrame), CVPixelBufferGetHeight(cameraFrame));
        CGSize scaledImageSizeToFitOnGPU = [GPUImageOpenGLESContext sizeThatFitsWithinATextureForSize:sizeOfPhoto];
       report_memory(@"Middle 1");
        if (!CGSizeEqualToSize(sizeOfPhoto, scaledImageSizeToFitOnGPU))
        {
            CMSampleBufferRef sampleBuffer;
            GPUImageCreateResizedSampleBuffer(cameraFrame, scaledImageSizeToFitOnGPU, &sampleBuffer);
       report_memory(@"Middle 2");

            dispatch_semaphore_signal(frameRenderingSemaphore);
           [self captureOutput:photoOutput didOutputSampleBuffer:sampleBuffer fromConnection:[[photoOutput connections] objectAtIndex:0]];
            dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);
            CFRelease(sampleBuffer);
       report_memory(@"Middle 2.2");

        }
        else
        {
       report_memory(@"Middle 3");

            dispatch_semaphore_signal(frameRenderingSemaphore);
            [self captureOutput:photoOutput didOutputSampleBuffer:imageSampleBuffer fromConnection:[[photoOutput connections] objectAtIndex:0]];
           report_memory(@"Middle 3.2");

            dispatch_semaphore_wait(frameRenderingSemaphore, DISPATCH_TIME_FOREVER);
        }
        report_memory(@"After filter processing");

        UIImage *filteredPhoto = [finalFilterInChain imageFromCurrentlyProcessedOutput];
        dispatch_semaphore_signal(frameRenderingSemaphore);

        block(filteredPhoto, error);        
    }];

    return;
}

I've never run into a situation like this before where I've felt so helpless with a memory problem.

[manjeettbz]

how many filters you are apply? in my case its happening with crop and saturation filter chain. i did not try others.

[iamcam]

I'm using the crop filter on every one so I can get square output. In my app I've subclassed Group Filter so I could create my own custom sets. The chain generally looks like this:

Source -> Crop (squre) -> subclassedGroup (for base) ->subclassedGroup (for overlay) -> Output View/File. Both of the subclassedGroup filters are independent and dynamic based either on user choice, or image input. Since the crop filter does not allow scaling, I may have to add one more filter at the beginning: Source -> ScaleDownFilter -> subclassedGroup -> etc

Each of my subclassedGroup filters only as 1-2 filters on average, though it may get more complex as I start really looking at what kinds of effects I'd like.

[iamcam]

I should also add, I've had Source -> fullsizeCropFilter going the whole time in parallel so I would have access to the original image. Removing that helps quite a bit (one less full-resoution image). I can take several photos in a row so long as I don't trigger my takePhoto: method too often. To cut down on memory, I'm using the buffer output pre-capture, at low resolution, to display to the user until they are done making adjustments. Meanwhile a clean image is saved to disk for final rendering at a later time. It's not my ideal way of approaching this, but it seems to be the only thing that gets me close.

Back to the tmp file. Inside of the [stilLCamera capturePhotoAs... method, do this, and it seems to help usability, if anything:

 dispatch_async(queue, ^{
            @autoreleasepool {
            report_memory(@"Beginning save queue");
            NSString *tmpPath = [NSTemporaryDirectory() stringByAppendingPathComponent:@"tmpBigImg"];
            [UIImagePNGRepresentation(processedImage) writeToFile:tmpPath atomically:YES];
            NSLog(@"Saved");
            report_memory(@"End save queue");

            }
        });

It's basically putting the tmp file save action in its own thread so the app can move on and not have to wait for this relatively slow operation. This way the user can still interact with the app & on the small buffer image in memory without deadlocking. I have no idea if @autoreleasepool does anything here.

Now the next thing to deal with:: Queuing up these asynchronous saves. If I rapidly take one photo after another, memory starts to climb rapidly. My best guess is that the full image data processedImage from the capture block, has to be kept in memory until it is saved, at which point it gets freed. If I stack 2 or 3 together, i can pretty much guarantee memory usage will skyrocket and the app will crash, as seen in the Instruments Allocations tool, the moment before it crashed:

If I give it enough time between shots, it seems to be fine enough to not crash. If that's what it comes down to, it shouldn't be hard to create a delegate method that calls back to the UIButton I use for capture and disable it until the queue shortens. I think Instagram does something like this - I noticed you can take two photos in rapid succession before it produces a spinner and makes you wait to re-take the photo. Golly, what a bunch of hoops we have to jump through; if only these phones had more ram.

So.... Here we have (or at least I have) somewhat of a work-around, but it's not ideal. I'm wondering if @BradLarson has any ideas about memory usage in the framework that could be buttoned up a little bit. It might be a nice option to have direct buffer-to-disk method that bypasses the whole UIImage memory hog. I don't know if we can pull data out of the buffer in chunks, but I think NSData can be written in blocks as to avoid loading the entire file in memory all at once.

[dmitric]

Hey @iamcam fwiw i also notice the same thing. All my code is here https://github.com/gobackspaces/DLCImagePickerController

I do the same thing, live preview (which has no problem) -> capture crop to uiimage -> add previous filter chain -> save, but my crash generally comes when i try to load the new image into a GPUImage when using a more complex filter (group filters, but not basic ones).

[iamcam]

@dmitric Interesting. I don't know if the @autorelease pool helps, but it's worth trying. Also, I end up resetting my entire chain when I switch from live to photo sources. That may or may not help; I ended up coming up with that approach some time ago when I was figuring out how to get around a framework bug.

[dmitric]

@iamcam are you saying that you capture directly from the stillCamera and then reapply everything? Right now I was doing up to the cropFilter to make it square, and then reapplying the rest.

[dmitric]

@iamcam that autorelease pool actually solved it! Right now all my live filtering and capture works, now just need to figure out how to get Group Filters to work properly on the GPUImagePicture, and figure out a nice way to have image overlays on pictures. By any chance do you know how to do overlays that obey opacity?

[manjeettbz]

i am testing it more, but seems like autorelease pool is working. i will post results soon. thanks @dmitric

[iamcam]

@dmitric I capture the output of the crop filter - to a UIImage, then use that as the basis for further filter changes. In order to use two-input filters: Search around - there is at least one ticket discussing how to generally do it - you basically add one input as a target at index 0, and the other input at index 1.

[dmitric]

@iamcam thanks! Will try it out. Thanks for being helpful

[iamcam]

How's it working out for you? I still see crashes from time to time, but it's vastly improved. I also decreased the resolution a bit, which probably helped reduce memory pressure more than anything.

[rromanchuk]

i am running into a similar reproducible issue. When I capture as photo to the first crop filter in order to save an original photo, about 50% of the time, when trying to apply the last effects filter to the image, my didReceiveMemoryWarning gets called and the last filter never gets ran.

self.camera capturePhotoAsImageProcessedUpToFilter:self.croppedFilter withCompletionHandler:^(UIImage *processedImage, NSError *error){
        //forceProcessingAtSize isn't working :(
        self.croppedImageFromCamera = [processedImage resizedImage:CGSizeMake(640.0, 640.0) interpolationQuality:kCGInterpolationHigh];

        self.previewImageView.image = [[(GPUImageFilterGroup *)self.selectedFilter terminalFilter] imageByFilteringImage:self.croppedImageFromCamera];
        [self.camera stopCameraCapture];
        UIImageWriteToSavedPhotosAlbum(self.previewImageView.image, self, nil, nil);
    }];

however, just running the entire filter chain at once, it works fine.

self.camera capturePhotoAsImageProcessedUpToFilter:self.selectedFilter withCompletionHandler:^(UIImage *processedImage, NSError *error){
        //forceProcessingAtSize isn't working :(
        self.croppedImageFromCamera = [processedImage resizedImage:CGSizeMake(640.0, 640.0) interpolationQuality:kCGInterpolationHigh];
        [self.camera stopCameraCapture];
        UIImageWriteToSavedPhotosAlbum(self.previewImageView.image, self, nil, nil);
    }];

I have attempted many things in order to release memory within the block so i can get the last filter to run without running out of memory to no avail.

[rromanchuk]

@iamcam @dmitric Here is how i solved my issue, I create a Group filter with croppedFilter->EffectsFilter, i run capturePhotoAsImageProcessedUpToFilter up to the cropped filter, i save the cropped original and that's it. I don't try to do anything else in the completion handler. Immediately after the block i call

[self performSelector:@selector(filter) withObject:nil afterDelay:2];

Inside the filter method I alloc init my last filter that i needed and any additional processing that was required. This is obviously not a "fix" but i have yet to have it crash. Just mock your live output with the groupfilter, capture up to crop, and do absolutely nothing inside the completion block. You can also "trick" the user that the filter is complete by just pausing the camera inside the block, it feels very responsive even with an artificial/arbitrary 2second delay + filter processing.

[rromanchuk]

@iamcam @dmitric I played around with this a bit more and i found another helpful "hack", you can double your available memory in capturePhotoAsImageProcessedUpToFilter: by removing all targets and removing the groupfilter and add ONLY the crop filter immediately after the user takes the picture.

Using the group filter with crop->brightness group I usually have about in capture photo: used: 361,402,368 free: 6,799,360 total: 368,201,728 in filter after block: used: 168,116,224 free: 212,832,256 total: 380,948,480

Using the group filter with crop->complicated filter I usually have about CRASH :(

Using only the crop filter in capture photo: used: 377,335,808 free: 16,863,232 total: 394,199,040 in filter after block: used: 183,918,592 free: 196,710,400 total: 380,628,992

For example....

- (IBAction)didTakePicture:(id)sender {

    GPUImageCropFilter *cropFilter = [[GPUImageCropFilter alloc] initWithCropRegion:CGRectMake(0.0, 0.125, 1.0, 0.75)];
    [self.camera removeAllTargets];
    [self.camera addTarget:cropFilter];
    [cropFilter prepareForImageCapture];
    [self.camera capturePhotoAsImageProcessedUpToFilter:cropFilter withCompletionHandler:^(UIImage *processedImage, NSError *error){
    //now do stuff with 2x+ avail memory
}

[rromanchuk]

I just did a bunch of more tests on the device and I am now consistently getting between 15MB-25MB of free memory inside capturePhoto when I ditch the groupfilter before capture. When using any combination of with group+crop I consistently get 0-6MB, which means doing any other image processing is futile in that range.

For reference this is going into capture with about 90-115MB free.

[sujal]

I made some of the changes that @rromanchuk recommended, but what made a huge different for my app (Which uses a lot of @dmitric's camera class, incidentally) was to reduce the size of the image getting presented in the GPUImagePicture used to preview edits.

What I did was reduce the square crop we had from 2448x2448 (the 4S's full res square) to 2048x2048 (after the picture is taken - basically, I take the image out of the crop filter, then resize it down if it's larger than 2048x2048, then hand that image to the GPUImagePicture class). I haven't seen it crash since. It looks like the texture size limits that are discussed in the GPUImage code for the 4S might be biting us here. Our app only needs 2048x2048 images, so I haven't investigated further, but in case it helps anyone else, just sharing this.

[dmitric]

@sujal Yes, I noticed that a popular photo sharing application does the same with regards to sizing it down.

[iamcam]

Wow, I had to take time off of this project for a few months, and now I'm just getting back into it. You guys are definitely on to something. I did a little bit of testing on my end and it seemed to work pretty well. Thanks for taking the time to figure it out.