Convert yolo failed using your way

[lumenghz]

Hi

I meet a problem when I convert yolo.weight (not tinyyolo) and hope you can help me.

Environment Python 3.5.2 + Python 2.7.13

pip2 list

coremltools (0.4.0)
funcsigs (1.0.2)
h5py (2.7.0)
Keras (2.0.4)
mock (2.0.0)
numpy (1.13.1)
pbr (3.1.1)
pip (9.0.1)
protobuf (3.3.0)
PyYAML (3.12)
scipy (0.19.1)
setuptools (32.1.0)
six (1.10.0)
tensorflow (1.0.1)
Theano (0.9.0)
wheel (0.29.0)

pip3 list

h5py (2.7.0)
Keras (2.0.4)
numpy (1.13.1)
olefile (0.44)
Pillow (4.2.1)
pip (8.1.2)
protobuf (3.3.0)
pydot-ng (1.0.0)
pyparsing (2.2.0)
PyYAML (3.12)
scipy (0.19.1)
setuptools (25.2.0)
six (1.10.0)
tensorflow (1.0.1)
Theano (0.9.0)
wheel (0.29.0)

Step 1 Convert yolo.weights to yolo.h5 success. And I've tested the .h5 can work normally.

Step 2 Convert yolo.h5 to coreml command: python2 coreml_yolo.py output:

Traceback (most recent call last):
  File "coreml_yolo.py", line 12, in <module>
    coreml_model = coremltools.converters.keras.convert('yad2k/model_data/yolo.h5')
  File "/usr/local/lib/python2.7/site-packages/coremltools/converters/keras/_keras_converter.py", line 477, in convert
    predicted_feature_name = predicted_feature_name)
  File "/usr/local/lib/python2.7/site-packages/coremltools/converters/keras/_keras2_converter.py", line 149, in _convert
    model = _keras.models.load_model(model)
  File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 240, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 304, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 2416, in from_config
    process_layer(layer_data)
  File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 2385, in process_layer
    custom_objects=custom_objects)
  File "/usr/local/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
    list(custom_objects.items())))
  File "/usr/local/lib/python2.7/site-packages/keras/layers/core.py", line 706, in from_config
    function = func_load(config['function'], globs=globs)
  File "/usr/local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 198, in func_load
    code = marshal.loads(code.encode('raw_unicode_escape'))
ValueError: bad marshal data (unknown type code)

I also tried Keras version 1.2.2, same problem.

[hollance]

Yes, the full YOLO model contains a type of layer that is currently not supported by Core ML.

[lumenghz]

@hollance Thanks for your reply 😄

And I have another problem. Something I downloaded TinyYOLO which have 80 classes and convert it to .mlmodel.

I record a gif.

Here is what I changed:

let lebels = [
"person", "bicycle", "car", "motorbike", "aeroplane", "bus", "train", "truck", "boat",
"traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat",
"dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack",
"umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball",
"kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket",
"bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple",
"sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake",
"chair", "sofa", "pottedplant", "bed", "diningtable", "toilet", "tvmonitor", "laptop",
"mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink",
"refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier",
"toothbrush"
]

let anchors: [Float] = [
0.738768 ,0.874946, 2.42204, 2.65704, 4.30971,
7.04493, 10.246, 4.59428, 12.6868, 11.8741
]

I set videoCapture.fps = 1

Did I made something wrong?

[lumenghz]

@hollance Do you have any ideas about it?

[hollance]

I haven't tried this version of the model myself. I first would check that the number of bounding boxes is correct (there seem to be quite a few of them), then what the coordinates of the boxes are. Maybe the anchors are wrong?

[lumenghz]

The number of bounding boxes is 10. (seems right?)

The .cfg of tiny-yolo and tiny-yolo-voc seems similar. So I guess I don't need change other properties except labels, anchors and model ?

And the anchors was copied from the generated tiny-yolo_anchors.txt. It should be right.

I've already ran test_yolo.py and the result seems correct. But it all mess when I change the model in iOS.

Do you mind to test it for me if you have time? Thanks in advance

[lumenghz]

Hi @hollance

I did some test today. This is the step below:

1. Add debugPrint at method computeBoundingBoxes after filter by confidence threshold.

   debugPrint("prediction count after filter by confidence ==> ", predictions.count)

2. Add debugPrint at method computeBoundingBoxes before return so the end of method become

let results = nonMaxSuppression(boxes: predictions, limit: YOLO.maxBoundingBoxes, threshold: iouThreshold)
debugPrint("result count after filter: ", results.count)
return results

3. Run & let camera face a object that can not be identified.

Result: When I ran your model (Tiny-YOLO-VOC) that have 20 classes, the two debugPrint always be 0 and that is the right result. When I ran the model (Tiny_YOLO) that have 80 classes. The first debugPrint always > 70 and the second one always = 10(limit box count)

The situation didn't become good even I change the confidenceThreshold to 0.8. So forget the anchors temporarily because this result is not expected.

The result seems good when I ran test_yolo.py after generated .h5. So I'm very doubt about the result I saw in iOS.

Do you have any ideas?

[lumenghz]

Hi @hollance

I think I found the reason after read all code. And that's my fault because I didn't read the code carefully before ask.

In method computeBoundingBoxes there is a const named numClasses, I need to change it to 80. And the colors should be changed also.

I'm sorry for disturbing you. Thanks to provided such a good example. 😄

[hollance]

Great, I'm happy it works now. :smile:

[pcmanik]

@hollance @lubeast I was trying to convert full yolo using yad2k too and have faced the same problem. The real issue is in the converted Keras model from Darknet and not that coreml don't support all of the layers.

@lubeast Have you made any progress in this area?

Simple script can prove that

from keras.models import load_model
model = load_model('yad2k/model_data/yolo-voc.h5')

[pchensoftware]

These comments might help:

https://github.com/fchollet/keras/issues/7440#issuecomment-343648667

https://github.com/allanzelener/YAD2K/issues/80#issuecomment-343685850