apple / coremltools

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.
https://coremltools.readme.io
BSD 3-Clause "New" or "Revised" License
4.4k stars 636 forks source link

NaN value when predict with model resnet #2025

Closed ThuyyTran closed 1 year ago

ThuyyTran commented 1 year ago

When I run coreml model in python result is good: {'var_840': array([[-8.15439941e+02, 2.88793579e+02, -3.83110474e+02, -8.95208740e+02, -3.53131561e+02, -3.65339783e+02, -4.94590851e+02, 6.24686813e+01, -5.92614822e+01, -9.67470627e+01, -4.30247498e+02, -9.27047348e+01, 2.19661942e+01, -2.96691345e+02, -4.26566772e+02........ But when I run on xcode so result look like: [-inf,inf,nan,-inf,nan,nan,nan,nan,nan,-inf,-inf,-inf,-inf,-inf,-inf,nan,-inf,-inf,nan,-inf,nan,nan,-inf,nan,-inf,-inf,-inf,nan,nan,nan,nan,nan,nan,nan,nan,nan,nan,-inf,nan,nan,nan,nan,-inf,nan,-inf .......

Step1: Convert Resnet50 to coreml:


import torch
import torchvision

# Load a pre-trained version of MobileNetV2 model.
torch_model = torchvision.models.resnet50(pretrained=True)
# Set the model in evaluation mode.
torch_model.eval()
# Trace the model with random data.
example_input = torch.rand(1, 3, 224, 224) 
traced_model = torch.jit.trace(torch_model, example_input)
out = traced_model(example_input)
# Download class labels in ImageNetLabel.txt.
# Set the image scale and bias for input image preprocessing.
import coremltools as ct
image_input = ct.ImageType(shape=example_input.shape,
                          )
# Using image_input in the inputs parameter:
# Convert to Core ML using the Unified Conversion API.
model = ct.convert(
    traced_model,
    inputs=[image_input],
    compute_units=ct.ComputeUnit.CPU_ONLY,
)
# Save the converted model.
model.save("resnettest.mlmodel")
# Print a confirmation message.
print('model converted and saved')

Step2: Test model coreml in python:


import coremltools as ct
import PIL
import numpy as np
# Load the model
model = ct.models.MLModel('/Users/ngoclh/Downloads/resnettest.mlmodel')
print(model)
img_path = "/Users/ngoclh/gitlocal/DetectCirtochApp/DetectCirtochApp/resources/image.jpg"
img = PIL.Image.open(img_path)
img = img.resize([224, 224], PIL.Image.ANTIALIAS)
coreml_out_dict = model.predict({"input_1" : img})
print(coreml_out_dict)

Step3: Test coreml model in Xcode:


func getFeature() {
    do {
        let deepLab = try VGG_emb.init() //mobilenet_emb.init()//cirtorch_emb.init()
        let image = UIImage(named: "image.jpg")
        let pixBuf = image!.pixelBuffer(width: 224, height: 224)!
        guard let output = try? deepLab.prediction(input_1: pixBuf) else {
            return
        }
        let names = output.featureNames
        print("ngoc names: ", names)
        for name in names {
            let feature = output.featureValue(for: name)
            print("ngoc feature: ", feature)
        }
    } catch {
        print(error)
    }
}

System environment (please complete the following information):

TobyRoseman commented 1 year ago

Coremltools 6.2 is quite old. Please try with the most recent version of coremltools.

ThuyyTran commented 1 year ago

Coremltools 6.2 is quite old. Please try with the most recent version of coremltools.

I used coremltools 6.3 but the output convert required .mlpackage format, not .mlmodel

TobyRoseman commented 1 year ago

If you want to save it in the .mlmodel format, you need to convert it for the neuralnetwork backend. This can be done directly by adding convert_to="neuralnetwork" to your ct.convert call. Although, a better approach would probably be to use the minimum_deployment_target argument. See ct.convert documentation for details.

Also note - that currently 7.0, not 6.3, is the latest version of coremltools.

ThuyyTran commented 1 year ago

If you want to save it in the .mlmodel format, you need to convert it for the neuralnetwork backend. This can be done directly by adding convert_to="neuralnetwork" to your ct.convert call. Although, a better approach would probably be to use the minimum_deployment_target argument. See ct.convert documentation for details.

Also note - that currently 7.0, not 6.3, is the latest version of coremltools.

@TobyRoseman I try update version coremltools to 7.0 and add arg convert_to="neuralnetwork" look like: ` torch_model = torchvision.models.resnet50(pretrained=True)

torch_model.eval() example_input = torch.rand(1, 3, 224, 224) traced_model = torch.jit.trace(torch_model, example_input) out = traced_model(example_input) import coremltools as ct image_input = ct.ImageType(name="input_1", shape=example_input.shape, ) model = ct.convert( traced_model, inputs=[image_input], compute_units=ct.ComputeUnit.CPU_ONLY,convert_to="neuralnetwork" ) model.save("restnet50_v7.mlmodel") ` But the result still looks bad @@ Screen Shot 2023-10-24 at 9 20 50 AM

TobyRoseman commented 1 year ago

Since your model works in Python, then this is not an issue with coremltools. This must be a problem with your swift code.

Take a look at the Classifying Images with Vision and Core ML tutorial. That should include the functionality you need.

If you still have questions after that, I suggest asking for help in the developer forum: https://developer.apple.com/forums/

ThuyyTran commented 1 year ago

@TobyRoseman Sorry but I don't know if this is the reason why my result is NaN or not. I fixed this error by changing the code at line 4377 in the path to the following "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/models/neural_network/builder.py", line 4373, in add_unary"

Old:

spec_layer_params.epsilon = epsilon
spec_layer_params.alpha = Alpha
spec_layer_params.shift = shift
spec_layer_params.scale = scale

Change:

if type(alpha) is numpy.ndarray:
      val_Alpha = alpha[0]
else:
    val_Alpha = alpha
spec_layer_params.epsilon = epsilon
spec_layer_params.alpha = val_Alpha
spec_layer_params.shift = shift
spec_layer_params.scale = scale

Error when I convert model

Traceback (most recent call last):
  File "ConvertSolar2Coreml.py", line 218, in <module>
    coreml_model = ct.convert(traced_model, inputs=[input_tensor], source='pytorch',convert_to="neuralnetwork")
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py", line 564, in convert
    use_default_fp16_io=use_default_fp16_io,
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/converter.py", line 217, in _mil_convert
    **kwargs
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/converter.py", line 304, in mil_convert_to_proto
    out = backend_converter(prog, **kwargs)
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/converter.py", line 119, in __call__
    return load(*args, **kwargs)
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/backend/nn/load.py", line 264, in load
    prog.functions["main"].outputs,
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 49, in convert_ops
    mapper(const_context, builder, op)
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 1127, in pow
    _add_elementwise_binary(const_context, builder, op, "pow")
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 768, in _add_elementwise_binary
    alpha=op.y.val,
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/converters/mil/backend/nn/op_mapping.py", line 683, in _add_elementwise_unary
    **kwargs
  File "/home/anlab/anaconda3/envs/testconvertmodel/lib/python3.7/site-packages/coremltools/models/neural_network/builder.py", line 4373, in add_unary
    spec_layer_params.alpha = alpha
TypeError: array([3.2220182], dtype=float32) has type numpy.ndarray, but expected one of: int, float
TobyRoseman commented 1 year ago

@ThuyyTran - I'm confused. You say your change fixed the error, but you also say you can no longer convert the model. How is the issue fixed if you can't even convert your model?

ThuyyTran commented 1 year ago

@TobyRoseman Sorry for that, I mean in the model conversion step I got the error as above. I tried to fix the error by editing the code in the coremltools library and converted successfully. I mean I'm wondering if me trying to fix the code in the library is the reason why my result is nan @@

TobyRoseman commented 1 year ago

Since you can get correct prediction in Python, I don't think this is an issue with coremltools. I think the is an issue with your Swift code.

ThuyyTran commented 1 year ago

@TobyRoseman But can you explain to me the error I get when converting the model?

TobyRoseman commented 1 year ago

It most be related to your local change.

ThuyyTran commented 1 year ago

@TobyRoseman Editing the library directly like that won't cause any issues, will it?