SpectrumKit not working with screenshot images in iOS 13

[rahulvyas]

I have following version of spectrum SpectrumCore (1.1.0) SpectrumKit (1.1.0) mozjpeg (3.3.2) spectrum-folly (2019.01.21.00)

When I try to compress a screenshot I'm getting a plain white image. I'm not getting any error. I'm testing on iPhone XR iOS 13.1.2.

Attaching some input images and some screenshot of what I'm getting as output.

I'm not getting any kind of error when compressing these images just plain white image I'm getting

This is my image picking code

- (void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary *)info {
   NSString *mediaType = [info objectForKey:UIImagePickerControllerMediaType];
   if ([mediaType isEqualToString:@"public.image"]){
        UIImage *image = [info objectForKey:UIImagePickerControllerOriginalImage];
        UIImage *transcoded = [self transcodeImage:image];
   }
}

- (UIImage*)transcodeImage:(UIImage*)image {

    FSPSpectrum *spectrum = [[FSPSpectrum alloc] initWithPlugins:@[[FSPJpegPlugin new]]
    configuration:nil];

    FSPEncodeRequirement *encodeRequirement =
        [FSPEncodeRequirement encodeRequirementWithFormat:FSPEncodedImageFormat.jpeg
                                                     mode:FSPEncodeRequirementModeLossy
                                                  quality:70];

    FSPConfiguration * configuration = [[FSPConfiguration alloc]init];

    FSPTransformations *transformations = [FSPTransformations new];
    transformations.resizeRequirement =
        [[FSPResizeRequirement alloc] initWithMode:FSPResizeRequirementModeExactOrSmaller
                                        targetSize:CGSizeMake(1024, 1024)];

    FSPEncodeOptions *options =
        [FSPEncodeOptions encodeOptionsWithEncodeRequirement:encodeRequirement
                                             transformations:transformations
                                                    metadata:nil
                                               configuration:configuration
                         outputPixelSpecificationRequirement:nil];

    NSError *error;
    FSPResultData *result = [spectrum encodeImage:image options:options error:&error];
    if (result.result.didSucceed && result.data) {
        UIImage* image = [UIImage imageWithData:result.data];
        if (image) {
            return image;
        }
    }
    return image;
}

Does anyone knows any fix. I have to release the app on store and it's the only issue I'm stuck with.

I've tried same images on facebook and these are working perfectly.

Edit : Updating targetSize:CGSizeMake(2048, 2048) in FSPResizeRequirement produces the below image

Edit 2 : Updating targetSize:CGRectZero in FSPResizeRequirement always gives error and I'm not able to compress the image

Here is the error log

Printing description of error:
Error Domain=com.facebook.spectrum Code=255 "(null)" UserInfo={com.facebook.spectrum.error-name=scalingX > 0, com.facebook.spectrum.error-location=facebook::spectrum::core::proc::ScalingBlockImpl::ScalingBlockImpl(const image::pixel::Specification &, const image::Size &, const image::Size &):62}

Edit 3 : Same code works fine in iPhone 6 (iOS 12.4.6)

[rahulvyas]

@wizh @cuva @diegosanchezr can you look into it ?

[rahulvyas]

Are the contributors still active in this library ?

[zmroczek]

Hi @rahulvyas

We are sorry for the delay.

The issue is connected with the fact that the images are 16-bits per channel and we are able to reproduce it on your images, however we are getting into the error in this line https://github.com/facebookincubator/spectrum/blob/15fb5fdc5988310ffc20b9eb5195dff5f3c7c3cd/ios/SpectrumKit/SpectrumKit/Image/FSPImagePixelSpecification.mm#L240 so not sure why you are getting a white screen instead.

A way for you to mitigate this will be to use an image with 8-bit per channel instead.

We currently don't support images with different number of bits per component but we would be very happy to welcome contributions in form of PRs if you'd like to add that enhancement.

[rahulvyas]

@zmroczek Thanks for the heads Up. However if you read my whole thread I'm getting different results based on TargetSize as I've mentioned. Also sometimes I get exception but it's not what you're telling. I request you to please go through the whole question again. As far as PR's , I am not much into image manipulations and C++. It would be great if someone experienced work on it. Isn't there any active contributor who can fix this ?

[rahulvyas]

@zmroczek I wonder how Facebook able to compress the same image properly ? Isn't the Facebook iOS app uses spectrum itself ?

[zmroczek]

Hi @rahulvyas

First of all, I'd like to explain that we found why you are not getting an exception but a distorted result instead - it's because you're using an old version of Spectrum (1.1.0) which didn't have the code in place for throwing an error that I pointed out above. So if you switch to the newest version you will be able to catch the error as well instead of displaying those broken results.

The reason why those screenshots are causing problems might be connected to this specific iOS version saving screenshots as 16bits per channel instead of 8 and therefore you might have started getting the problem because of that.

Currently we are not supporting images with different number of bits per channel than 8 on Spectrum for iOS but we are investigating on fixing that.

[rahulvyas]

@zmroczek Here is my podfile and podfile.lock enteries

Podfile pod 'SpectrumKit/Plugins/Jpeg'

Podfile.lock

- spectrum-folly (2019.01.21.00)
  - SpectrumCore/Base (1.1.0):
    - spectrum-folly (~> 2019.01.21.00)
  - SpectrumCore/Plugins/Jpeg (1.1.0):
    - mozjpeg (= 3.3.2)
    - spectrum-folly (~> 2019.01.21.00)
    - SpectrumCore/Base (= 1.1.0)
  - SpectrumKit/Base (1.1.0):
    - spectrum-folly (~> 2019.01.21.00)
    - SpectrumCore/Base
  - SpectrumKit/Plugins/Jpeg (1.1.0):
    - spectrum-folly (~> 2019.01.21.00)
    - SpectrumCore/Plugins/Jpeg
    - SpectrumKit/Base (= 1.1.0)

Is there any specific version you want me to upgrade to ? If yes please suggest.

[AurelC2G]

Spectrum 1.2 was released back in June, and includes the check mentioned above.

[rahulvyas]

@AurelC2G, @zmroczek I wonder why I was getting different results based on the target size value ? Anyone have any idea about that ?

[zmroczek]

@rahulvyas I'm suspecting that this is due to bytes per pixel in image specification not being passed correctly for those images when we were not throwing an error in the older version of Spectrum you're using: https://github.com/facebookincubator/spectrum/blob/15fb5fdc5988310ffc20b9eb5195dff5f3c7c3cd/ios/SpectrumKit/SpectrumKit/Image/FSPImagePixelSpecification.mm#L209

Possibly this affects how the output image looked like but we need to dig deeper into that.

Thank you for providing this repro case, we will use it for investigating the solution for this enhancement. If you'd like to dig deeper nevertheless and contribute you are also very welcome to do that.

[rahulvyas]

@zmroczek I wonder how facebook able to compress the same image properly ? aren't the facebook iOS app uses spectrum itself ?

@zmroczek Anyone knows about this ?

[rahulvyas]

@rahulvyas I'm suspecting that this is due to bytes per pixel in image specification not being passed correctly for those images when we were not throwing an error in the older version of Spectrum you're using: https://github.com/facebookincubator/spectrum/blob/15fb5fdc5988310ffc20b9eb5195dff5f3c7c3cd/ios/SpectrumKit/SpectrumKit/Image/FSPImagePixelSpecification.mm#L209

Possibly this affects how the output image looked like but we need to dig deeper into that.

Thank you for providing this repro case, we will use it for investigating the solution for this enhancement. If you'd like to dig deeper nevertheless and contribute you are also very welcome to do that.

Any updates on this ?

[zmroczek]

Hi @rahulvyas,

We're still looking into adding the support for images with other values in bits per component than 8 in the encoding function.

In the meantime to get the correct output, you could try using transcodeImage function rather than encodeImage by passing the input file directly there rather than decoded bitmap (as in the encoding function you're currently using). Let us know if this fixes the issue for you.

[rahulvyas]

Thanks @zmroczek. I'll change the code as you've suggested. Thanks for your help.

[EwoutGelling]

Hello! For anyone also having this issue I fixed this problem by converting the Images to have 8bit depth, using the Apple Accelerate Framework. This code works on IOS 13 and up, don't forget to import Accelerate.

 private static func converTo8BitDepth(image: UIImage) -> UIImage? {

        guard (image.cgImage?.bitsPerPixel != 32) else {
            return image
        }

        guard let cgImage = image.cgImage,
              let sourceImageFormat = vImage_CGImageFormat(cgImage: cgImage),
              let destImageFormat = vImage_CGImageFormat(bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: CGColorSpaceCreateDeviceRGB(), bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.first.rawValue), renderingIntent: .defaultIntent) else {
                  // Handle error.
                  return nil
              }

        guard let sourceBuffer = try? vImage_Buffer(cgImage: cgImage),
              var rgbDestBuffer = try? vImage_Buffer(width: Int(sourceBuffer.width), height: Int(sourceBuffer.height), bitsPerPixel: destImageFormat.bitsPerPixel) else {
                  // Handle error.
                  return nil
              }
        defer {
            sourceBuffer.free()
            rgbDestBuffer.free()
        }

        do {
            let toRgbConverter = try vImageConverter.make(sourceFormat: sourceImageFormat, destinationFormat: destImageFormat)

            try toRgbConverter.convert(source: sourceBuffer, destination: &rgbDestBuffer)
        } catch {
            // Handle error.
            print(error.localizedDescription)
        }
        if let cgImage = try? rgbDestBuffer.createCGImage(format: destImageFormat) {
            return UIImage(cgImage: cgImage)
        } else {
            // Handle error.
            return nil
        }
    }

[rahul-sysquare]

This code works on IOS 13 and up, don't forget to import Accelerate.

is it merged into main?