The conversion doesn't seem to work for me

pawartur commented 7 years ago

Hi!

I have a darknet model that is based on tiny-yolo-voc and finds license plates pretty accurately (checked that many times with darknet's detector command).

Your conversion mechanism works pretty well for your model (just like you write in readme).

However, when I apply your conversion process to my darknet model, the resulting coreml model doesn't seem to be able to recognise anything.

Would you be kind enough to help me in any way?

hollance commented 7 years ago

Do you get an error message?

Also see this: http://machinethink.net/blog/help-core-ml-gives-wrong-output/

pawartur commented 7 years ago

Thanks again for such a quick response.

Actually, I wasn't specific enough describing what I'd done.

Here it is:

I modify the standard tiny-yolo-voc.cfg file and modify to change the number of classes to 1
I have ~400 pics of car license plates
I run darknet detector train (...) with my modified tiny-yolo-voc-1c.cfg file and a proper .data file that points to my ~400 pics as training pics and another ~50 pics as test pics
After 10000 round I get a model with error margin < 1.0
I run darknet detector test (...) with the resulting .weights file on my test images
the result of the test is amazing :)
I use your yad2k.py script with my tiny-yolo-voc-1c.cfg and my resulting .weigths file to convert the darknet model to keras
I run your test_yolo.py script to check, if the conversion was successful. I use the same test images as in the case of darknet detector test (...)
No positive matches are made with test_yolo.py
I convert the converted .h5 keras model to .mlmodel format using your coreml.py script
I use the resulting .mlmodel file in my iOS app
No car plates are detected properly

So it seems for me that the problem in my case is in the step that converts darknet to keras. I've gone through the yad2k.py script and didn't find anything that could be specific to the standard tiny-yolo-voc case that you used it for, but maybe there is something...

What do you think?

I'm quite new to ML and Vision, so maybe I messed sth. up. I would appreciate your help a lot :).

hollance commented 7 years ago

The yad2k script isn't mine so I don't really know the ins and outs, but if the test_yolo script already doesn't work then it's likely the conversion to Keras did not go right -- or the input images passed into the model in test_yolo are in a different format than the format you trained darknet on.

pawartur commented 7 years ago

Do you know the author of yad2k script?

hollance commented 7 years ago

The link is in the readme.

zhenkaiwang commented 7 years ago

@pawartur Did you find a solution? I have met the exactly the same issue as yours... My model works on the original darknet but doesn't work after converted to Keras model. Still couldn't find the reason for this.

pawartur commented 7 years ago

Hi!

Unfortunately not. And I need to leave this for now, will probably get back to solving this in september.

Best of luck with solving this, I will probably ping you in a few weeks :-).

On 8 Aug 2017, at 02:17, zhenkaiwang notifications@github.com wrote:

@pawartur Did you find a solution? I have met the exactly the same issue as yours... My model works on the original darknet but doesn't work after converted to Keras model. Still couldn't find the reason for this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

richard-giantrobot commented 6 years ago

I'm having the exact same issue. The Darknet to Keras converter works fine when using the weights from https://pjreddie.com/darknet/yolo/. But not with our own weights. It breaks down on the exact same spot as with @pawartur (Darknet to Keras converter).
Also noticed that the pb file one can create from darkflow works perfectly on Android and Windows devices in our case. I've been looking at YAD2k to see what needs to change to fix this.
If anybody finds anything pls post a message.

zhenkaiwang commented 6 years ago

I solve my problem by directly retraining a keras model through a modified retrain_yolo.py.

On 2017年9月11日, at 07:33, Richard Chirino notifications@github.com wrote:

I'm having the exact same issue. The Darknet to Keras converter works fine when using the weights from https://pjreddie.com/darknet/yolo/. But not with our own weights. It breaks down on the exact same spot as with @pawartur (Darknet to Keras converter). Also noticed that the pb file one can create from darkflow works perfectly on Android and Windows devices in our case. I've been looking at YAD2k to see what needs to change to fix this. If anybody finds anything pls post a message.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

pawartur commented 6 years ago

That's an interesting solution, @zhenkaiwang :).

I haven't solved the problem yet, needed to leave it for now.

sarpsolakoglu commented 6 years ago

I also trained darknet with tiny-yolo-voc configuration for only one object. It works before conversion and detects the objects as expected but I cannot get it to work after converting to Keras. Did anybody solve how to make this work with a weight file from training a custom object and not the pre-trained weights that is provided?

oishi89 commented 6 years ago

@zhenkaiwang could you please give some guidelines how to retraining the model

helloniklas commented 6 years ago

@zhenkaiwang are you able to share your retrain_yolo.py please? Or nay guidance on how to retrain this model on different classes and images would be great.

richard-giantrobot commented 6 years ago

I was finally able to use my custom darknet model with CoreML by doing the following:

I used darkflow (https://github.com/thtrieu/darkflow) to convert my model to a Tensorflow protobuf. Then I used Apple and Google's new Tensorflow to CoreML converter (https://developers.googleblog.com/2017/12/announcing-core-ml-support.html) to convert the model into an mlmodel. This conversion went well and allowed me to use my custom darknet model (tiny-yolo) in CoreML.

Important to keep in mind that my yolo model used BGR for the pixel order and expected a value from 0-1 so I also had to set the scale and is_bgr to true. My conversion script ended up looking like this:

import tfcoreml as tf_converter
tf_converter.convert(tf_model_path = 'yolo.pb',
                     mlmodel_path = 'yolo.mlmodel',
                     output_feature_names = ['output:0'],
                     is_bgr = True,
                     input_name_shape_dict = {'input:0' : [1, 416, 416, 3]},
                     image_input_names = ['input:0'],
                     image_scale = 1 / 255.0)

oishi89 commented 6 years ago

@richard-giantrobot great job ! could you let me know how many Fps your tiny yolo coreml is on real device thanks

richard-giantrobot commented 6 years ago

@oishi89

I'm getting 30 fps on my iPhone X, and around 5 fps on an iPhone 6 Plus. I don't currently have other devices to test against.

bmgdev commented 6 years ago

@richard-giantrobot this is very promising! having same issues you are. 2 questions:

1) were you able to use Matthjis's ios code in this repo with little modification with the returned mlmodel from TF Convert? (obv class changes, input names etc)

2) in TFConvert (coreml), is that block above that you used ALL that you used? or did you have to do any stripping of ops? I struggled greatly using those TF scripts for my SSD-MobileNets model, made me strip and use Preproccessor/sub as my input name (which CoreML hates).

This is the most promising news ive heard yet! Thank you! b

helloniklas commented 6 years ago

@bmgdev Did you try apple's new open source Turi Create. Which runs a tiny yolo behind the scenes, and is very easy to retrain and export for iOS.

bmgdev commented 6 years ago

WOW @helloniklas! this is huge!! thank you so much for the link! now to test this puppy out... have you had successes with it yet?

oishi89 commented 6 years ago

@helloniklas thank so much for your sharing. have you had a chance to test a model trained by Turi on real devices ? As you mentioned the tiny yolo is being used under the hook so i think the performance is similar to tiny yolo model. It would be around 5 - 6 fps for IPhone 6 / 5se

helloniklas commented 6 years ago

@bmgdev @oishi89 yes works great. Managed to train my custom model on 1 class with only 30 images. And it’s still very accurate running at 22 FPS. They recommend at least 200 images for better results. But so far works really well. And super easy to use and train with. Lots of tools to visualise the test data to see what’s going on as well.

bmgdev commented 6 years ago

@helloniklas ill have to try that next, looking forward to diving in. I JUST this second got my model converted by way of Darkflow->TFCoreml and working in-app, so pretty happy. Turi next!

oishi89 commented 6 years ago

@helloniklas what kind of iphone you were testing ? is it iphone X or earlier ? 22fps is impressive

oishi89 commented 6 years ago

@helloniklas if you don’t mind could you share a quick video for showing the accuracy and performance. oishi89@gmail.com

helloniklas commented 6 years ago

@oishi89 it's running on an iPhone X, same fps as when I used a keras converted tiny yolo.

richard-giantrobot commented 6 years ago

@bmgdev

were you able to use Matthjis's ios code in this repo with little modification with the returned mlmodel from TF Convert? (obv class changes, input names etc)

All I had to change was the number of classes, the labels and the name of the input and output tensors which darkflow automatically changes to input and output (in the predict function, I'm not running it trough the Vision API right now, it works but it looks slower).

in TFConvert (coreml), is that block above that you used ALL that you used? or did you have to do any stripping of ops? I struggled greatly using those TF scripts for my SSD-MobileNets model, made me strip and use Preproccessor/sub as my input name (which CoreML hates).

Yeah that was all I used, after loading the model in my app it ran correctly right away. I have no experience with SSD-MobileNets so I don't know what difference that makes.

RGravity commented 6 years ago

@richard-giantrobot I got my Model Converted to CoreML using your Methods, but now I can't seem to get it to run in the TinyYOLO-CoreML

Image of the Error

Any ideas?

richard-giantrobot commented 6 years ago

@RGravity

Yeah that was what I meant with I needed to change the input and output names. Should be like this (if you used darkflow):

if let output = try? model.prediction(input__0: image) {
    return computeBoundingBoxes(features: output.output__0)
} else {
    return []
}

RGravity commented 6 years ago

@richard-giantrobot Thanks thats it 👍

tgwagner commented 6 years ago

@bmgdev @richard-giantrobot can you please share steps on how you converted your darknet tiny-yolo model from tensorflow (pb) to coreml?

I was able to convert my tiny-yolo model from darknet to tensorflow but can't seem to convert from tensorflow (.pb) to .mlmodel

I get the following error:

Traceback (most recent call last): File "converttest1.py", line 6, in image_input_names = ['input:0']) File "/Users/tgwagner/fresh/tf-coreml/tfcoreml/_tf_coreml_converter.py", line 483, in convert predicted_probabilities_output=predicted_probabilities_output) File "/Users/tgwagner/fresh/tf-coreml/tfcoreml/_tf_coreml_converter.py", line 175, in _convert_pb_to_mlmodel ('Output feature cannot be a placeholder') AssertionError: Output feature cannot be a placeholder

when I run:

import tfcoreml as tf_converter
tf_converter.convert(tf_model_path = '/users/fresh/darkflow/built_graph/tiny-yolo-NFPA.pb',
                     mlmodel_path = 'my_model.mlmodel',
                     output_feature_names = 'output:0',
                     input_name_shape_dict = {'input:0':[1, 416, 416, 3]},
                     image_input_names = ['input:0'])

I also ran summarize on the *.pb model and here's the output:

=/users/fresh/darkflow/built_graph/tiny-yolo-NFPA.pb
Found 1 possible inputs: (name=input, type=float(1), shape=[?,416,416,3]) 
No variables spotted.
Found 1 possible outputs: (name=output, op=Identity) 
Found 15770590 (15.77M) const parameters, 0 (0) variable parameters, and 0 control_edges
Op types used: 59 Const, 16 Mul, 9 Conv2D, 9 BiasAdd, 9 Pad, 8 Sub, 8 Maximum, 8 RealDiv, 6 MaxPool, 1 Placeholder, 1 Identity, 1 NoOp

Thanks!

bmgdev commented 6 years ago

@tgwagner you need to flow your graph. take your darknet tiny yolo model to darkflow, then you can run those same commands in tf_converter

https://github.com/thtrieu/darkflow

look under: "Save the built graph to a protobuf file (.pb)"

this should be the only command you need:

flow --model lex-yolo.cfg --load lex-yolo.weights —savepb

then to tf convert with 'import tfcoreml as tf_converter'...

you're very close :) b

tgwagner commented 6 years ago

@bmgdev thanks for your quick reply! Yes, I've actually already flowed my model, that's where I got my *.pb file (originally weights). My issue is converting the pb file into the mlmodel file.

I used flow --model tiny-yolo-nfpa.cfg --load tiny-yolo-nfpa_6100.weights -savepb

How did you convert pb to mlmodel?

bmgdev commented 6 years ago

@tgwagner only other step I had in addition to yours is image scale.

import tfcoreml as tf_converter
tf_converter.convert(tf_model_path = 'lex-yolo.pb',
                     mlmodel_path = 'lex-yolo.mlmodel',
                     output_feature_names = ['output:0'],
                     input_name_shape_dict = {'input:0' : [1, 416, 416, 3]},
                     image_input_names = ['input:0'],
                     image_scale = 1 / 255.0)

based on your error "Output feature cannot be a placeholder", I assume this is an error in training?

Overview on placeholders here: https://learningtensorflow.com/lesson4/

and in tf convert script where its catching that placeholder:

  # run through all placeholders
  for op in OPS:
    output_names = set([compat.as_bytes(x.name) for x in op.outputs])
    if op.type == 'Placeholder':
      # Handle placeholders -- all placeholders are inputs
      assert not filter(output_names.__contains__, output_feature_names), \
          ('Output feature cannot be a placeholder')
.
.
.

Hopefully that gives you a little more to poke at.... b

bmgdev commented 6 years ago

actually @tgwagner, i think I see your error: check this line:

output_feature_names = 'output:0',

should be:

output_feature_names = ['output:0'],

:)

tgwagner commented 6 years ago

@bmgdev I did a clean install of coremltools and tfcoreml (using python 2.7 vs 3.6) it worked. Thanks for your help investigating. As an aside, I can't believe how fragmented this toolchain is.

bmgdev commented 6 years ago

@tgwagner I was just about to mention trying py3 instead of py2 - happy you got it. guys like @hollance gave me a pointer or two when I first started out, the more we can help each other the better.

The fragmentation has only begun. Am now trying to slim my model to a set of ops that TF Lite supports so I can start doing cross platform here. If you want more fun, try getting TF Mobile building on iOS... highly dependent on specific versions and quite fragile. Oh boy the fun never stops.

Best of luck on your journey.

tgwagner commented 6 years ago

@bmgdev cheers buddy, I hear you. I'm sure we'll be chatting a bunch more!

saikatbsk commented 6 years ago

Had similar problems, the solution from @richard-giantrobot worked. Used darkflow to convert my custom yolo model to tf protobuf. then Used tfcoreml to convert to coreml model. One thing I have noticed that the darkflow model is less accurate, though. Darknet to keras using yad2k is much more accurate.

akuma308 commented 6 years ago

@richard-giantrobot @saikatbsk i have created the .pb file from darknet (it does produce a .meta) file too. when i am using @richard-giantrobot code for converting it into mlmodel , it is indeed creating a mlmodel file, basically a json i quess, but i have read that it should create shrad files too(assuming it is the weights file for the graph) . Can you please guide us that just a .*model file is enough . I will be using tensorflow.js.

JiangkunLiu commented 6 years ago

@richard-giantrobot @saikatbsk I am trying to get the tiny-yolo-voc to work on my iPhone X. I am using this project: https://github.com/hollance/YOLO-CoreML-MPSNNGraph. It gives me ridiculous bounding box predictions. I tried to convert the Darknet to .pb and then convert to core ml using tfcoreml, I still get similar wrong predictions. Anybody has an idea what might be the reason it dint work for me?

saikatbsk commented 6 years ago

@JiangkunLiu @akuma308 I was successful in converting a darknet YOLOv2 model to .pb and then to .mlmodel. But it didn't work for TinyYOLO. I had followed the instructions mentioned here: https://github.com/thtrieu/darkflow

hollance / YOLO-CoreML-MPSNNGraph

The conversion doesn't seem to work for me #7