CoreML as a feature extractor

[osianSmith]

❓Question

Hi all I am trying to use CoreML as a feature extractor (from sound analysis). I've split up by model as I said in issue #589 however when I try to get Xcode to start the model I receive this error:

Invalid model, classification models must have 'predictedProbabilitesName' and 'predictedFeaturesKey' properties Unable to prepare request: Invalid model, classification models must have 'predictedProbabilitesName' and 'predictedFeaturesKey' properties

Currently, the model consists of sound analysis and the neural network (we've removed the GLM classifier). Is there any way we can get CoreML to return a MultiArray (Float32 12288)?

Thank you

Osian

[califrench]

Hi @osianSmith ,

Your model currently defines itself as a neuralNetworkClassifier. You can see that by loading your model and inspecting its contents:

import coremltools
from coremltools import *
model = models.MLModel('path_to.mlmodel')
spec = model.get_spec()
print(spec.neuralNetworkClassifier) # This may take a while and be very long
print(len(spec.neuralNetworkClassifier.layers)) # a quicker approach would be to count the layers

The neural network classifier provides some conveniences towards the end by returning a dictionary of probabilities as well as a top class label.

What you want in your case is a non-classifier neural network.

You will need to copy all the neural network layers from the neuralNetworkClassifier to the neuralNetwork property of your model like so

print(spec.neuralNetwork.layers) # This should return an empty list []
# Now let's copy over the layers
layers = spec.neuralNetworkClassifier.layers
spec.neuralNetwork.SetInParent()
spec.neuralNetwork.layers.extend(layers)
# You'll also need to adjust your outputs to match the last layer of your neural network
# First, remove the existing outputs
while len(spec.description.output):
    spec.description.output.pop()

# Now add the new multi array output
spec.description.output.add()
output = spec.description.output[0]
# Configure your output
output.name = "classLabelProbs" # This needs to match exactly the output name of your output layer
# You can get it by looking at the last item in `layers`, for example on MobileNet I get the following
layers[-1]
# Out[29]: 
# name: "prob"
# input: "fc7"
# output: "classLabelProbs"
# softmax {
# }
# As a description for convenience
output.shortDescription = "The features extracted by the model"
# Set the type to multi array and its shape (here the shape is [1, 1000])
output.type.multiArrayType.SetInParent()
output.type.multiArrayType.shape.append(1)
output.type.multiArrayType.shape.append(1000)
# Finally, you'll want to save your new spec as a new model
utils.save_spec(spec, 'new_model.mlmodel')

Hope this helps!

[osianSmith]

Hello @califrench Thanks for your help, however, when looking at the model's layers[-1] I get an out of bounds error on the original model that CreateML has produced and also on a model that had been split. Is this likely that I'm not accessing a sub-model and if I am how would I do this?

Thanks again

[califrench]

Hi @osianSmith, I'm not 100% sure about the model you have. Is it a pipeline or is it now a single neural network? If it's a pipeline then you would need to access the layers with spec.pipeline.models[index].neuralNetworkClassifier.layers.

[osianSmith]

It's a pipeline

spec.pipeline.models[0].neuralNetworkClassifier.layers (instead of the index as index is not defined in my code) and it's complaining it's out of range, Is there likely an Issue with my model? @califrench

[osianSmith]

Hi all I managed to grab the output of the by grabbing the feature extractor by: featureExtratorModel = pipeline_spec.models[1]

output name here was:

output.name = "vggishFeature"

I then put the output as = output.type.multiArrayType.shape.append(1) output.type.multiArrayType.shape.append(12288)

However, Xcode returns this error: validator error: Description of multiarray feature 'vggishFeature' has an invalid or unspecified dataType. It must be specified as DOUBLE, FLOAT32 or INT32

I'm going to try to see if I can fix this before possibly opening another question but if anyone has any idea I would appreciate the help.

Thank you

Osian

[califrench]

Hi @osianSmith ,

You have one minor step left, which is to define the datatype on your multiArray feature.

Right next to where you define the shape you can define the type as well.

The values for the types are described in the Core ML Model Specification

message ArrayFeatureType {

enum ArrayDataType {
        INVALID_ARRAY_DATA_TYPE = 0;
        FLOAT32 = 65568; // 0x10000 | 32
        DOUBLE = 65600;  // 0x10000 | 64
        INT32 = 131104;  // 0x20000 | 32
    }

output.type.multiArrayType.dataType = 65568 # For FLOAT32

There's a way to import the specification from the coremltools instead of package instead of hard-coding the values but I find this method quicker.

You would do that this way:

import coremltools.proto.FeatureTypes_pb2
output.type.multiArrayType.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.FLOAT32

Hope this helps!

[osianSmith]

Hey, thanks! That fixed that error however I am now getting this error: validator error: Type of pipeline output 'vggishFeature' does not match type produced in pipeline input.

This is my last layer: name: "flatten" input: "pool4_output" output: "vggishFeature" flatten { mode: CHANNEL_LAST }

And where I am setting my output: `output.type.multiArrayType.SetInParent() output.type.multiArrayType.shape.append(0) output.type.multiArrayType.shape.append(12288)

sets the data type for our mutiArray feature

output.type.multiArrayType.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.FLOAT32`

Could it be that I have either need to remove one more layer or is the output of this not a vggish feature or can't CoreML know how to cope with this or is it something completely different?

Thanks a lot :)

[TobyRoseman]

It sounds like the original issue has been resolved. So I'm going to close this issue. If there are additional problem, please create new issues with complete steps to reproduce the problem.