Converted classification model returns invalid confidence scores (e.g. negative numbers and Nan)

🐞Describe the bug

I converted a TensorFlow graph into a mlmodel. I can correctly deploy the mlmodel file except for the fact that the values it returns do not make any sense. It's a classification model supposed to return confidence scores in [0,1], but it returns arbitrary large and/or small numbers. Once, I even got a NaN value.

The original TensorFlow graph works just fine. It feels like maybe the preprocessing isn't applied correctly? I have configured the preprocessing during the conversion though.

To Reproduce

This is how I converted the model:


    bias = -mean / std
    image_input = ct.ImageType(shape=(1, input_size, input_size, 3,),
                               bias=[bias, bias, bias],
                               scale=1 / std,
                               name=input_feature_name,
                               )

    ml_model = ct.convert(
        model=model_path,
        inputs=[image_input],
        outputs=[output_feature_name],
        minimum_deployment_target=ct.target.iOS13,
    )

    ml_model.save(output_path)

I'm sorry, I don't have model file which I'm allowed to share. However, I could reproduce this with multiple models (e.g. Inceptionv3 and MobileNetv2).

System environment (please complete the following information):

coremltools version 4.0:
OS (e.g., MacOS, Linux): Ubuntu 18
How you install python (anaconda, virtualenv, system): virtualenv
python version (e.g. 3.7): 3.7
any other relevant information:
- TensorFlow: 1.15

Additional context

My mlmodel integration within my iOS app isn't a new one. I originally set it up to target iOS 11 along with the following successful setup:

tfcoreml 1.0
TensorFlow 1.14
coremltools 3.1 (not sure about this, wasn't added explicitly, but used by tfcoreml)

That's working as it should. It works with older versions of the same graphs, which are failing now, too. However, before retraining my model I migrated from TensorFlow 1.14 to 1.15 and so I couldn't use that setup anymore.

As an intermediate solution I also tried to use tfcoreml 1.1 with coremltools 3.4 to target iOS 13. It suffers the same issue as the finally used setup of coremltools 4.0.

I got no clue how to continue debugging this. Any help is appreciated very much!

apple / coremltools