Activity classifier deployment error on iPhone SE, iOS 11.2.5

[srikris]

Following a conversation from https://github.com/apple/turicreate/issues/146#issuecomment-368526896 with @LennartOlsen

I am using the example supplied from https://github.com/apple/turicreate/blob/master/userguide/activity_classifier/export_coreml.md my only difference is that i supply the data from an external sensor not from the internal CMAccelerometerData

Running the application on my MacBook pro 14,1 running macOS 10.13.3 it works fine (as a native app), though running the exact same framework on my iPhone SE, iOS 11.2.5 gives

validateComputeFunctionArguments:852: failed assertion `Compute Function(cnnConvArray_8xIn_8xOut_1_1): The pixel format (MTLPixelFormatRGBA32Float) of the texture (name:) bound at index 0 is incompatible with the data type (MTLDataTypeHalf) of the texture parameter (srcArray [[texture(0)]]). MTLPixelFormatRGBA32Float is compatible with the data type(s) ( float ).' Ive tried tracing the error but I cant seem to figure out where it goes wrong. My predictor is https://github.com/LennartOlsen/SensorTagRaider/blob/bd6fbff9d582095c10c8dfa07b6c16d6b509b84d/SensorTagRaider/Controllers/Predictor/Predictor.swift#L59

[LennartOlsen]

@srikris it could look like a GPU error with Core ML on iOS devices, the problem persists across the following devices :

iPad Pro 9.7 ; ios 11.0
iPhone SE    ; ios 11.2.5 & 11.2.6
iPhone 7     ; ios 11.2

Unfortunately i am not able to replicate it on the MacOS emulator as it does not support the Bluetooth Device i am using to collect the data.

Let me know if you need anything

[igiloh]

Hi @LennartOlsen, sorry that you're having this problem

A few things that come to mind:

1. Are you initializing your dataArray at some point? It should be initialized (probably to zeros) before the first calls to predict() - so if any of the sensors' readings are missing, you would still have some numerical value in the array.
2. It seems that you're only casting Accelerometer data to Float. Are the other sensors guaranteed to be floats? Can you try casting these as well?
3. Better yet - can you please try casting all of them to Double? (it seems that your Measurement is defined as Double anyway). The exported model probably expects Doubles as input (you can check by double-clicking the mlmodel file in Xcode).
4. Could it be that you sometimes call PerformPrediction() with only part of the sensors in dataEntry? For example with a Gyroscope measurement only, but without new Magnetometer reading? If so - the other sensors must be attended to somehow (filled with some default values). Otherwise - you keep calling some of the sensors with their previous readings, and in case of the first call - with some uninitialized values.

Could you please try running a simpler model with just one of the sensors, just to rule out all of the possible errors above?
Could you also please try using your code (all 3 sensors), on the same devices, but with the built-in sensors?

Last but not least - if none of the above helps, could you please try to record some data from your sensor and send it in some format (csv etc)? I'd like to try to reproduce your problem, using your app, in order to debug it.

[LennartOlsen]

@igiloh Ive tried to simply add dummy data to every element of the data prediction array as shown here :

https://gist.github.com/LennartOlsen/bb1d8462fc6473602caa6c4616f4d381

Still gives the same error :

validateComputeFunctionArguments:852: failed assertion `Compute Function(cnnConvArray_8xIn_8xOut_1_1): The pixel format (MTLPixelFormatRGBA32Float) of the texture (name:<null>) bound at index 0 is incompatible with the data type (MTLDataTypeHalf) of the texture parameter (srcArray [[texture(0)]]). MTLPixelFormatRGBA32Float is compatible with the data type(s) (
    float
).'

My model looks like

To my best of knowledge the error looks like a type casting error related to the Metal Framework, MTLDataTypeHalf which is basically a 16bit Float that is not compatible with MTLPixelFormatRGBA32Float which is a set of 32-bit Floats. But forgive me im very new to the the entire ecosystem regarding CoreML

Give me time and i will try to create a very simplified example for you guys, i very much appreciate the help and quick feedback 👍

I tried a different case using another model (which was created for testing purposes), this one takes a vector of [1,1,9] as seen below.

It produces the same error, although the function has a different signature this time around cnnConvArray_8xIn_8xOut_1_2 rather than cnnConvArray_8xIn_8xOut_1_1

validateComputeFunctionArguments:852: failed assertion `Compute Function(cnnConvArray_8xIn_8xOut_1_2): The pixel format (MTLPixelFormatRGBA32Float) of the texture (name:<null>) bound at index 0 is incompatible with the data type (MTLDataTypeHalf) of the texture parameter (srcArray [[texture(0)]]). MTLPixelFormatRGBA32Float is compatible with the data type(s) (
    float
).'

[igiloh]

Hi @LennartOlsen

To my best of knowledge the error looks like a type casting error related to the Metal Framework

You're probably right - I'm simply trying to rule out some obvious caveats in the way the model is called before diving deep down into Core ML's metal implementation. Core ML export of activity_classification models was tested on a range of iOS devices - so I'm trying to figure out if it's some specific to your code or a more general error.

Thanks for the input! Will keep investigating.

[LennartOlsen]

@igiloh looking at #146 i see that the comment by gustavla suggested the following

In Xcode: Product -> Scheme -> Edit Scheme... -> Options -> Metal API Validation -> Set to Disabled

I worked for me on the iPhone SE, furthering that this i properly a hardware/software communication issue.

For now i can live with the solution as i dont plan on using this commercially, but i dont think it can be considered solved.

Let me know if theres anything i can do to help :+1:

[igiloh]

@LennartOlsen, I'm glad to hear that you're now unblocked. Like you, I agree that this isn't a good permanent solution. If you could please send some easy steps\data to easily reproduce the problem - we'll be glad to dive deeper and find a proper fix.

[LennartOlsen]

I will try my very best!

[LennartOlsen]

@igiloh ok, so Ive tried to create a very simple solution some interesting points came up.

• Application https://github.com/LennartOlsen/CoreMLTuricreate/tree/master/CoreMLTuricreate
• Jupyter Notebook (not necessary) https://github.com/LennartOlsen/SensorTagData

As you notice the application is littered with a simple Metal rendering mechanism, that is due to the fact that my devices did not seem to activate Metal API - And not activating the Metal API would not provoke an error.

My original application also had a simple Metal requirements in rendering a small 3d figure, i removed it and the errors went away.

[znation]

This should now be fixed in the latest iOS 11 releases. Thanks for your patience.