How to print model layers shape

fantasyRqg commented 4 years ago

❓Question

Use neural_network.NeuralNetworkBuilder built a model spec.
when use spec new MLModel produce an error: "compiler error: Espresso exception: "Invalid blob shape": generic_elementwise_kernel: cannot broadcast [1, 14280, 2, 1, 1] and [14280, 2, 1, 1, 1]".

print spec just some layer connection. no shape info.

I'm using coremltools build a model decode bbox like sdd-caffe does.

System Information

Mac OS
coremltools 4.0b2

Code

    fd_mbox_priorbox = np.load('fd_mbox_priorbox.npy')
    num_prior_ = int(fd_mbox_priorbox.shape[2] / 4)

    # MLMultiArray inputs of neural networks must have 1 or 3 dimensions.
    # We only have 2, so add an unused dimension of size one at the back.
    input_features = [("scores", datatypes.Array(num_classes + 1, num_prior_, 1)),
                      ("boxes", datatypes.Array(4, num_prior_, 1))]

    # The outputs of the decoder model should match the inputs of the next
    # model in the pipeline, NonMaximumSuppression. This expects the number
    # of bounding boxes in the first dimension.
    output_features = [
        # ("decoded_bbox_out", datatypes.Array(4, num_prior_, 1)),
        ("confidence", datatypes.Array(num_prior_, num_classes + 1, 1)),
        ("coordinates", datatypes.Array(num_prior_, 4, 1))
    ]

    builder = neural_network.NeuralNetworkBuilder(input_features,
                                                  output_features,
                                                  disable_rank5_shape_mapping=True
                                                  )

    # decode_bbox_center_x = prior_variance[0] * bbox.xmin() * prior_width + prior_center_x;
    # decode_bbox_center_y = prior_variance[1] * bbox.ymin() * prior_height + prior_center_y;
    # decode_bbox_width = exp(prior_variance[2] * bbox.xmax()) * prior_width;
    # decode_bbox_height = exp(prior_variance[3] * bbox.ymax()) * prior_height;

    fd_mbox_priorbox = fd_mbox_priorbox.reshape([4, -1])
    prior_bbox = fd_mbox_priorbox[:, :num_prior_]
    prior_variances = fd_mbox_priorbox[:, num_prior_:]

    xmin, ymin, xmax, ymax = prior_bbox
    prior_width = xmax - xmin
    prior_height = ymax - ymin
    prior_center_x = (xmin + xmax) / 2.
    prior_center_y = (ymin + ymax) / 2.

    prior_wh = np.stack([prior_width, prior_height])
    prior_xy = np.stack([prior_center_x, prior_center_y])

    prior_variances = np.expand_dims(prior_variances, axis=-1)
    prior_wh = np.expand_dims(prior_wh, axis=-1)
    prior_xy = np.expand_dims(prior_xy, axis=-1)

    builder.add_load_constant('prior_wh',
                              output_name='prior_wh_out',
                              constant_value=prior_wh,
                              shape=prior_wh.shape
                              )

    builder.add_load_constant('prior_xy',
                              output_name='prior_xy_out',
                              constant_value=prior_xy,
                              shape=prior_xy.shape)

    builder.add_load_constant('prior_variances',
                              output_name='prior_variances_out',
                              constant_value=prior_variances,
                              shape=prior_variances.shape)
    print(prior_variances.shape)
    print(prior_wh.shape)
    print(prior_xy.shape)

    builder.add_split('split_variances',
                      input_name='prior_variances_out',
                      output_names=['split_variances_xy', 'split_variances_wh'],  # (2,num_prior_,1) , (2,num_prior_,1)
                      # num_splits=2,
                      # axis=-3
                      )

    builder.add_split('split_box',
                      input_name='boxes',
                      output_names=['split_box_min_out', 'split_box_max_out'],  # (2,num_prior_,1) , (2,num_prior_,1)
                      # num_splits=2,
                      # axis=-3
                      )

    # prior_variance[0] * bbox.xmin() * prior_width
    # prior_variance[1] * bbox.ymin() * prior_height
    builder.add_elementwise('bbox_xy_m',
                            input_names=['split_variances_xy', 'split_box_min_out', 'prior_wh_out'],
                            output_name='bbox_xy_m_out',  # (2,num_prior_,1)
                            mode='MULTIPLY'
                            )
    # prior_variance[0] * bbox.xmin() * prior_width + prior_center_x
    # prior_variance[1] * bbox.ymin() * prior_height + prior_center_y
    builder.add_elementwise('bbox_xy_a',
                            input_names=['bbox_xy_m_out', 'prior_xy_out'],
                            output_name='bbox_xy_a_out',
                            mode='ADD'
                            )

    # prior_variance[2] * bbox.xmax()
    # prior_variance[3] * bbox.ymax()
    builder.add_elementwise('bbox_wh_m',
                            input_names=['split_variances_wh', 'split_box_max_out'],
                            output_name='bbox_wh_m_out',
                            mode='MULTIPLY'
                            )

    # exp(prior_variance[2] * bbox.xmax())
    # exp(prior_variance[3] * bbox.ymax())
    builder.add_unary('bbox_wh_e',
                      input_name='bbox_wh_m_out',
                      output_name='bbox_wh_e_out',
                      mode='exp'
                      )

    # exp(prior_variance[2] * bbox.xmax()) * prior_width
    # exp(prior_variance[3] * bbox.ymax()) * prior_height
    builder.add_elementwise('bbox_wh_m_m',
                            input_names=['bbox_wh_e_out', 'prior_wh_out'],
                            output_name='bbox_wh_m_m_out',
                            mode='MULTIPLY'
                            )

    builder.add_elementwise('decoded_bbox',
                            input_names=['bbox_xy_a_out', 'bbox_wh_m_m_out'],
                            output_name='decoded_bbox_out',
                            mode='CONCAT'
                            )

    builder.add_nms('nms',
                    input_names=['decoded_bbox_out', 'scores'],
                    output_names=['coordinates', 'confidence', 'nms_indices', 'nms_number'],
                    iou_threshold=0.3,
                    score_threshold=0.05,
                    max_boxes=200
                    )
    builder.inspect_layers(verbose=True)

    nms_model = ct.models.MLModel(builder.spec)
    nms_model.save('nn.mlmodel')

    print("success")

leovinus2001 commented 4 years ago

Not sure whether it helps but have you tried logging like in #818 to gain further insights ?

fantasyRqg commented 4 years ago

use CoreML::Model load spec with some layer unsupport modification. I can print output blobs shape.

all seems OK .

when use Apple MLModel::compileModelAtURL still produce error error: Error compiling model: "compiler error: Espresso exception: "Invalid state": generic_split_kernel: Number of top blobs in split kernel does not evenly divide width axis."

error message is so vague. And no way show verbose info about layers output/input shape when compile model.😭

fantasyRqg commented 4 years ago

builder.inspect_layers message here. asking for help

/Users/rqg/Playground/coremltools/envs/coremltools-py3.7/bin/python3.7 /Users/rqg/PycharmProjects/CoreML/Convertfd.py
4.0b3
(4, 14280, 1)
(2, 14280, 1)
(2, 14280, 1)
[Id: 11], Name: nms (Type: NonMaximumSuppression)
          Updatable: False
          Input blobs: ['decoded_bbox_out', 'scores']
          Output blobs: ['coordinates', 'confidence', 'nms_indices', 'nms_number']
          Parameters: 
              iouThreshold = 0.30000001192092896
              scoreThreshold = 0.05000000074505806
              maxBoxes = 200
              perClassSuppression = False
[Id: 10], Name: decoded_bbox (Type: concat)
          Updatable: False
          Input blobs: ['bbox_xy_a_out', 'bbox_wh_m_m_out']
          Output blobs: ['decoded_bbox_out']
          Parameters: 
              sequenceConcat = False
[Id: 9], Name: bbox_wh_m_m (Type: multiply)
          Updatable: False
          Input blobs: ['bbox_wh_e_out', 'prior_wh_out']
          Output blobs: ['bbox_wh_m_m_out']
          Parameters: 
              alpha = 0.0
[Id: 8], Name: bbox_wh_e (Type: unary)
          Updatable: False
          Input blobs: ['bbox_wh_m_out']
          Output blobs: ['bbox_wh_e_out']
          Parameters: 
              type = 4
              alpha = 1.0
              epsilon = 9.999999974752427e-07
              shift = 0.0
              scale = 1.0
[Id: 7], Name: bbox_wh_m (Type: multiply)
          Updatable: False
          Input blobs: ['split_variances_wh', 'split_box_max_out']
          Output blobs: ['bbox_wh_m_out']
          Parameters: 
              alpha = 0.0
[Id: 6], Name: bbox_xy_a (Type: add)
          Updatable: False
          Input blobs: ['bbox_xy_m_out', 'prior_xy_out']
          Output blobs: ['bbox_xy_a_out']
          Parameters: 
              alpha = 0.0
[Id: 5], Name: bbox_xy_m (Type: multiply)
          Updatable: False
          Input blobs: ['split_variances_xy', 'split_box_min_out', 'prior_wh_out']
          Output blobs: ['bbox_xy_m_out']
          Parameters: 
              alpha = 0.0
[Id: 4], Name: split_box (Type: split)
          Updatable: False
          Input blobs: ['boxes']
          Output blobs: ['split_box_min_out', 'split_box_max_out']
          Parameters: 
              nOutputs = 2
[Id: 3], Name: split_variances (Type: split)
          Updatable: False
          Input blobs: ['prior_variances_out']
          Output blobs: ['split_variances_xy', 'split_variances_wh']
          Parameters: 
              nOutputs = 2
[Id: 2], Name: prior_variances (Type: loadConstant)
          Updatable: False
          Input blobs: []
          Output blobs: ['prior_variances_out']
          Parameters: 
              shape = [4, 14280, 1]
              data = (57120 floatValues)
[Id: 1], Name: prior_xy (Type: loadConstant)
          Updatable: False
          Input blobs: []
          Output blobs: ['prior_xy_out']
          Parameters: 
              shape = [2, 14280, 1]
              data = (28560 floatValues)
[Id: 0], Name: prior_wh (Type: loadConstant)
          Updatable: False
          Input blobs: []
          Output blobs: ['prior_wh_out']
          Parameters: 
              shape = [2, 14280, 1]
              data = (28560 floatValues)
/Users/rqg/Playground/coremltools/coremltools/models/model.py:119: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  Espresso exception: "Invalid state": generic_split_kernel: Number of top blobs in split kernel does not evenly divide width axis.".
  RuntimeWarning,

franva commented 2 years ago

I got the same issue as well.

Any update?? @jakesabathia2 @fantasyRqg

SichangHe commented 1 year ago

Was hunting down for a solution when I got here. One way to do it seems to be rewriting this Swift function in Python:

https://github.com/danielnugraha/flower-ios/blob/21cf129026861d6253030c512647918f7aa6d373/FlowerIOS/Utils/CoreMLHelpers/MLModelInspect.swift#L23-L47

apple / coremltools