converting 'cat' operation

Mayor2305 commented 2 months ago

🐞Describing the bug

Make sure to only create an issue here for bugs in the coremltools Python package. If this is a bug with the Core ML Framework or Xcode, please submit your bug here: https://developer.apple.com/bug-reporting/
Provide a clear and consise description of the bug. While converting from detectron2 model to coreml, got the following error:

ValueError: dtypes needs to be a list/tuple of at least 1 element

Stack Trace

ERROR - converting 'cat' op (located at: 'roi_heads/box_pooler'):

Converting PyTorch Frontend ==> MIL Ops: 75%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉ | 1426/1891 [00:00<00:00, 1685.47 ops/s] Traceback (most recent call last): File "/home/user/Desktop/drive/segmentation/model/torch2coreml.py", line 12, in mlmodel = ct.converters.convert( File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/_converters_entry.py", line 581, in convert mlmodel = mil_convert( File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 188, in mil_convert return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, kwargs) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 212, in _mil_convert proto, mil_program = mil_convert_to_proto( File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 288, in mil_convert_to_proto prog = frontend_converter(model, kwargs) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/converter.py", line 108, in call return load(*args, **kwargs) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 82, in load return _perform_torch_convert(converter, debug) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 116, in _perform_torch_convert prog = converter.convert() File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 581, in convert convert_nodes(self.context, self.graph) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 86, in convert_nodes raise e # re-raise exception File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 81, in convert_nodes convert_single_node(context, node) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 134, in convert_single_node add_op(context, node) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 2115, in cat values=promote_input_dtypes(xs), axis=axis, name=node.name File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/mil/ops/defs/_utils.py", line 452, in promote_input_dtypes promoted_dtype = promote_dtypes([var.dtype for var in input_vars]) File "/home/user/anaconda3/envs/coreml/lib/python3.10/site-packages/coremltools/converters/mil/mil/types/type_mapping.py", line 264, in promote_dtypes raise ValueError("dtypes needs to be a list/tuple of at least 1 element") ValueError: dtypes needs to be a list/tuple of at least 1 element

System environment (please complete the following information):

coremltools version: 7.2
OS (e.g. MacOS version or Linux type): Ubuntu 20.04.6 LTS
Any other relevant version information (e.g. PyTorch or TensorFlow version): torch==2.2.0, detectron2==0.6

Additional context

also initially i was getting a KeyError: 4 for the full operation, to counter that, i copied code from the latest beta release for the full operation. the copied code:

@register_torch_op
def full(context, node):
    inputs = _get_inputs(context, node, min_expected=2)

    size = inputs[0]

    # dtype could be torch.dtype or an integer that maps to a numpy.dtype
    dtype = None
    if len(inputs) < 3 or inputs[2] is None:
        dtype = np.float32
    elif isinstance(inputs[2].val, torch.dtype):
        dtype = NUM_TO_NUMPY_DTYPE[TORCH_DTYPE_TO_NUM[inputs[2].val]]
    elif isinstance(inputs[2].val, (int, np.generic)):
        dtype = NUM_TO_NUMPY_DTYPE[inputs[2].val]
    else:
        raise ValueError(f"unsupported type {type(inputs[2].val)}.")

    val = dtype(inputs[1].val)

    result = _make_fill_op(size, val, node.name)
    context.add(result)

junpeiz commented 2 months ago

Could you provide a code snippet for reproducing this? Thanks!

Mayor2305 commented 2 months ago

Here is a colab notebook to reproduce the issue. https://colab.research.google.com/drive/191HxvCwjIFlUIT3gOBPl7YGUZNnUZTJi?usp=sharing,

there are two issues:

when you run this notebook, you will get an error KeyError: 4. I soved this error by changing the code for full operation at coremltools/converters/mil/frontend/torch/ops.py (took it from coremltool 8 beta) to:

@register_torch_op
def full(context, node):
inputs = _get_inputs(context, node, min_expected=2)

size = inputs[0]

# dtype could be torch.dtype or an integer that maps to a numpy.dtype
dtype = None
if len(inputs) < 3 or inputs[2] is None:
    dtype = np.float32
elif isinstance(inputs[2].val, torch.dtype):
    dtype = NUM_TO_NUMPY_DTYPE[TORCH_DTYPE_TO_NUM[inputs[2].val]]
elif isinstance(inputs[2].val, (int, np.generic)):
    dtype = NUM_TO_NUMPY_DTYPE[inputs[2].val]
else:
    raise ValueError(f"unsupported type {type(inputs[2].val)}.")

val = dtype(inputs[1].val)

result = _make_fill_op(size, val, node.name)
context.add(result)

PLEASE RESTART THE NOTEBOOK AFTER THIS CHANGE.

Then I got the error for cat operation: ValueError: dtypes needs to be a list/tuple of at least 1 element

I hope this will help reproduce. Please let me know if you need anything else.

Mayor2305 commented 1 month ago

Any updates yet?

junpeiz commented 1 month ago

Thank you for providing the detailed steps to reproduce! I can reproduce the issue. We are investigating on the root cause of this bug.

apple / coremltools