Fix bug classifying png images

[jaalger]

I am writing a visual recognition application that uses VisualRecognition.classify in order to classify images. I have configured my Swift environment and have been able to classify images when including a URL from the internet:

Example:

 visualRecognition.classify("http://blog.fashionsealhealthcare.com/sites/default/files/styles/blog_image_display/public/field/image/blogs/ibm_watson.png", failure: visualRecongnitionDidFail){
            classifiedImages in
            print(classifiedImages)
}

I have now created an application that uses the camera and photo library to allow users to take photos and have them classified. I am running into issues when passing along an NSURL from the device to the VisualRecognition service though.

Example:

I create and save an image to the local device and then pass the NSURL to a simple classify function:

func classifyImage(imagePath: NSURL){
                visualRecognition.classify(imagePath, failure: visualRecongnitionDidFail){
            classifiedImages in
            print(classifiedImages)
        }

    }

The NSURL that I have been attempting to pass is as follows:

file:///var/mobile/Containers/Data/Application/C30D220F-699C-441E-A29B-6320391C96BD/Documents/temptest.png

when running the classify command with this NSURL I receive the following error:

Error Domain=com.alamofire.error Code=-6004 "Data could not be serialized. Failed to parse JSON response. No error information was provided during serialization." UserInfo={NSLocalizedFailureReason=Data could not be serialized. Failed to parse JSON response. No error information was provided during serialization.}

I have even attempted to include the NSURL directly into the classify call as I have done with the working external URL, but still run into the same error. Would really like to see how to use a local image from the device in order to classify it successfully

[glennrfisher]

Hi @jaalger. Just want to let you know that I'm looking into this. I'm digging through the SDK and our tests now to try and figure out what might be going wrong.

[jaalger]

Thanks for looking into it @glennrfisher. Looking at the tests it seems that the images are stored locally in the application, but I am attempting to grab images from the users camera and/or photo library.

[glennrfisher]

Yup! I understand. My plan was to run the tests to confirm that they pass, then create a sample application to test the camera / photo library. But I'm seeing some failures with the tests, so I think that's the best place to investigate. I'm hoping that the same problem is causing the failing tests and the issue that you ran into.

[glennrfisher]

For what it's worth, here's the URL of the car image that we use in the tests: file:///Users/glennfisher/Library/Developer/Xcode/DerivedData/WatsonDeveloperCloud-dcsqzcwzspfrpocltsdjxgrayzcl/Build/Products/Debug-iphonesimulator/VisualRecognitionV3Tests.xctest/car.png.

That's similar to the URL you included in the issue. My hunch is that it's not an issue loading the file, but something wrong with our request. (It looks like the format of the JSON parameters file changed, for example.)

[jaalger]

@glennrfisher I also ran into a similar issue with the detectFaces call. I can open another issue if needed, but with a simple detectFaces call it fails with a missing JSON dictionary value before I can actually catch it. For example:

func detectFaces(){    

 visualRecognition.detectFaces("https://upload.wikimedia.org/wikipedia/commons/7/7e/Thomas_J_Watson_Sr.jpg", failure: visualRecongnitionDidFail){
           detectedFaces in
           print(detectedFaces)
       }
   }

The above call throws the failure visualRecognitionDidFail and throws the following error:

Error Domain=com.alamofire.error Code=-6004 "Data could not be serialized. Failed to parse JSON response. The key (max) was not found in the JSON dictionary." UserInfo={NSLocalizedFailureReason=Data could not be serialized. Failed to parse JSON response. The key (max) was not found in the JSON dictionary.}

It seems like some of the returned values may be null, but I cannot even perform null checks since the call fails and never makes it to the detectedFaces. Could be related to the JSON format issue we are seeing above.

[glennrfisher]

I'm seeing some very strange behavior and haven't been able to figure out why. Our tests receive a successful response when classifying a .zip file of .jpg images or when classifying a single .jpg. But when I try classifying a .png, it fails. I'm trying to figure out why.

[jaalger]

I just verified that I could use the classify function with a jpg instead of a png that I grabbed from the camera roll. Same seems to be true with the detect faces function. Seems it is the .png files that are failing

[glennrfisher]

Okay, glad to know you're not dead in the water. Sounds like the .jpg images are working okay for you?

We'll have to figure out this issue with .png images. They are supported just fine by the service, so there must be something wrong with our requests. Perhaps there is a bug introduced in newer versions of Alamofire? Whatever the case may be, it may take some time for us to track it down.

[glennrfisher]

Okay, I've been poking around a bit and found an explanation for what's happening.

[glennrfisher]

The Problem

When uploading PNG images to Visual Recognition, we receive the following error from the service: "An undefined server error occured."

Solution

This problem is caused by the Xcode build process. When building for iOS (or the iOS Simulator), Xcode optimizes PNG images for quick rendering on the iPhone. The optimized image, however, is not a valid PNG format, causing the service's image processing library to fail.

To fix the problem, we need to disable Xcode's build-time optimizations. This can either be done on a file-by-file basis, or for all PNG images in the project:

Disable Optimization for a Single PNG File:

• Select the image file in the Navigator
• Show the Utilities Drawer then show the File Inspector
• Change the "Type" from "Default - PNG Image" to "Data"

Disable Optimization for All PNG Files:

• Select your project in the Navigator
• Select your build target in the targets list
• Select the Build Settings tab
• Select "All" to show all build settings
• Search for "compress"
• Set "Compress PNG Files" to "No"
• Set "Remove Text Metadata From PNG Files" to "No"

Explanation

During the build process, Xcode optimizes PNG images for the iPhone. It byte-swaps the red and blue octets then premultiplies the alpha channel so that the PNG can be quickly loaded and rendered by the iPhone. It also modifies the metadata stored in the image's header. The optimized image no longer adheres to the PNG specification, making it an invalid PNG file that cannot be recognized by the Visual Recognition service.

Our tests used to pass, however, because earlier versions of Xcode only optimized PNG images when building for a device. When building for the Simulator--which we used to execute our tests--the PNG images were not optimized but directly copied into the application bundle. That means this problem always plagued the SDK, but only caused tests to fail with more recent Xcode builds.

References:

• Blog Post: iPhone "Optimized" PNGs
• Apple FAQ Question
• StackOverflow Answer

Further Investigation

The Visual Recognition class currently accepts images as a URL, however, it may be more convenient for users to pass UIImage arguments. If we modify the class to accept a UIImage, then we can use the UIImagePNGRepresentation(_:) function to access the image's data in PNG format.

In that case, it's possible that our tests would work even with Xcode's optimized PNG images. That is, the UIImagePNGRepresentation(_:) function may reverse the optimizations. Then we can update the tests to load a UIImage instead of a URL, and use UIImagePNGRepresentation(_:) to get a valid PNG that can be sent to the Visual Recognition service.

See #353 for more information about migrating to UIImage instead of URL.