search

apple / coremltools

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.
https://coremltools.readme.io
BSD 3-Clause "New" or "Revised" License
4.45k stars 645 forks source link

Compiler Error for DeepLabv3 wrapper model when specifying output format as ImageType: main output 'output' must be a tensor of type Float32 or Float16 #2380

Closed himanshunaidu closed 1 week ago

himanshunaidu commented 3 weeks ago

🐞Describing the bug

Greetings, I am trying to convert a DeepLabv3 model into an mlpackage file. For that, I am using the official Apple documentation and building up from there. Link: https://apple.github.io/coremltools/docs-guides/source/convert-a-pytorch-segmentation-model.html

The default DeepLabv3 model from the tutorial works, however, we want a model that can be used as a VNCoreMLModel. The tutorial model accepts MLMultiArray, and we want a model that accepts Image. It would also be necessary for it to output an image, for performance as well as code convenience.

Thus, I am changing the input and output formats in the ct.convert function.

The input format change works as long as I do the normalization. Regarding the output, the model by default returns a tensor of dimensions (1 x num_classes x height x width). This output contains the class probabilities of every pixel in the input image. Thus, when I change the model output format, I need to also take the torch.argmax so that I can directly get an ImageType, where each pixel is the class with the highest probability for the corresponding input pixel. When I use the torch.argmax, I manage to save the model, but I get a runtimewarning, which talks about a compiler error. While I am able to save the model despite this error, I cannot use it in Xcode, as it says the following:

There was a problem decoding this Core ML document
missingMetadataField(named: "inputSchema")

Stack Trace

/opt/anaconda3/envs/deeplabv3/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. warnings.warn( /opt/anaconda3/envs/deeplabv3/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=DeepLabV3_ResNet101_Weights.COCO_WITH_VOC_LABELS_V1. You can also use weights=DeepLabV3_ResNet101_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Using cache found in /Users/tcat/.cache/torch/hub/pytorch_vision_v0.6.0 When both 'convert_to' and 'minimum_deployment_target' not specified, 'convert_to' is set to "mlprogram" and 'minimum_deployment_target' is set to ct.target.iOS15 (which is same as ct.target.macOS12). Note: the model will not run on systems older than iOS15/macOS12/watchOS8/tvOS15. In order to make your model run on older system, please set the 'minimum_deployment_target' to iOS14/iOS13. Details please see the link: https://apple.github.io/coremltools/docs-guides/source/target-conversion-formats.html Converting PyTorch Frontend ==> MIL Ops: 100%|▉| 847/848 [00:00<00:00, 3038.91 o Running MIL frontend_pytorch pipeline: 100%|█| 5/5 [00:00<00:00, 63.53 passes/s] Running MIL default pipeline: 100%|████████| 88/88 [00:04<00:00, 21.79 passes/s] Running MIL backend_mlprogram pipeline: 100%|█| 12/12 [00:00<00:00, 91.47 passes /opt/anaconda3/envs/deeplabv3/lib/python3.9/site-packages/coremltools/models/model.py:489: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Encountered an error while compiling a neural network model: validator error: main output 'output' must be a tensor of type Float32 or Float16". _warnings.warn(

To Reproduce

import coremltools as ct

model = torch.hub.load("pytorch/vision:v0.6.0", "deeplabv3_resnet101", pretrained=True).eval()

class WrappedDeeplabv3Resnet101(nn.Module):

def __init__(self):
    super(WrappedDeeplabv3Resnet101, self).__init__()
    self.model = torch.hub.load(
        "pytorch/vision:v0.6.0",
        "deeplabv3_resnet101",
        pretrained=True
    )
    self.model.eval()

def forward(self, input):
    res = self.model(input)["out"]
    return torch.argmax(res, dim=1, keepdim=True)

input_image = Image.open("images/cat_dog.jpg") preprocess = transforms.Compose([ transforms.ToTensor() ]) input_tensor = preprocess(input_image) input_batch = input_tensor.unsqueeze(0)

torch_model = WrappedDeeplabv3Resnet101().eval() traced_model = torch.jit.trace(torch_model, input_batch)

scale = 1/(0.226*255.0) bias = [- 0.485/(0.229) , - 0.456/(0.224), - 0.406/(0.225)]

image_input = ct.ImageType(name="input", shape=input_batch.shape, scale=scale, bias=bias) image_output = ct.ImageType( name="output", color_layout=ct.colorlayout.GRAYSCALE )

mlmodel_from_trace = ct.convert( traced_model, inputs=[image_input], outputs=[image_output] # MARK: This is what is causing the issue ) mlmodel_from_trace.save("SegmentationModel_no_metadata_from_trace.mlpackage")

load the model

mlmodel = ct.models.MLModel("SegmentationModel_no_metadata_from_trace.mlpackage")



## System environment (please complete the following information):
 - coremltools version: 8.0
 - OS (e.g. MacOS version or Linux type): Mac Ventura 13.5.2
 - Any other relevant version information (e.g. PyTorch or TensorFlow version): PyTorch 2.4.0
 - Here's my environment's yaml file: [deeplabv3.txt](https://github.com/user-attachments/files/17599750/deeplabv3.txt)

## Additional context
- Add anything else about the problem here that you want to share.
When I did this using a modified version of the following file, that uses [DeepLabV3Plus-Pytorch](https://github.com/VainF/DeepLabV3Plus-Pytorch.git), I don't get that error:
https://github.com/NaturalStupidlty/deeplabv3coreml/blob/main/from_deeplab.py
kinghchan commented 2 weeks ago

maybe try torch.argmax(res, dim=1, keepdim=True).float() ?

himanshunaidu commented 1 week ago

Hello @kinghchan , Apologies for not getting back on this. I actually did try the solution much after posting this, and this is what worked. Thank you so much.

I am just surprised that it was not needed for the code in this file: https://github.com/NaturalStupidlty/deeplabv3coreml/blob/main/from_deeplab.py