Using .pb file with iOS

[eweill]

I am trying to use a darknet/darkflow model with iOS. I have an application which can perform detection as well as a trained network from darknet. I used the .weights file from darknet to generate the .pb file using darkflow with the following command.

./flow --model cfg/yolo-new.cfg --load new.weights --savepb

When running the model on the iOS device, I am getting the same two recurring messages in the debug window.

Session created.
Graph Created.
Running model failed:Invalid argument: Session was not created with a graph before Run()!

Creating session.
Running model failed:Invalid argument: The first dimension of paddings must be the rank of inputs[4,2] [480,480,3]

I assume the first error has something to do with the second error; however, I may be incorrect. If anyone has any ideas about how to solve this or has successfully taken a model from darkflow and ported it to iOS, your experience would be greatly appreciated.

[abagshaw]

How are you pre-processing your images?

[eweill]

The code that I am using is an adapted form of ios_camera_object_detection. The piece of code that does the preprocessing (reading) of images, I believe, is as follows:

tensorflow::Tensor image_tensor(
      tensorflow::DT_FLOAT,
      tensorflow::TensorShape(
        {image_height, image_width, wanted_channels}));
  LOG(ERROR) << "Tensor Size: " << image_tensor.dim_size(0);
  auto image_tensor_mapped = image_tensor.tensor<float, 3>();
  tensorflow::uint8 *in = sourceStartAddr;
  float *out = image_tensor_mapped.data();
  for (int y = 0; y < image_height; ++y) {
    float *out_row = out + (y * image_width * wanted_channels);
    for (int x = 0; x < image_width; ++x) {
      tensorflow::uint8 *in_pixel =
          in + (y * image_width * image_channels) + (x * image_channels);
      float *out_pixel = out_row + (x * wanted_channels);
      for (int c = 0; c < wanted_channels; ++c) {
        out_pixel[c] = in_pixel[wanted_channels-c-1];
      }
    }
  }

I also tried modifying darkflow to output a .pbtxt file instead of the binary .pb file so that I could see exactly what the graph looked like. The following is where the error is coming from I think, I just don't know if darkflow is generating this by default or if I am doing something incorrectly:

node {
    name: "Pad/paddings"
    op: "Const"
    attr {
        key: "dtype"
        values {
            type: DT_INT32
        }
    }
    attr { 
        key: "value"
        value: {
            tensor {
                dype: DT_INT32
                tensor_shape {
                    dim {
                        size: 4
                    }
                    dim {
                        size: 2
                    }
                }
                tensor_content: "\000\000......"
            }
        }
    }
}

[abagshaw]

Try un-commenting https://github.com/thtrieu/darkflow/blob/master/net/build.py#L136 to test the .pb file you are creating to make sure darkflow has no trouble using it.

[thtrieu]

I did port several models to iOS successfully using this repo. Waiting for @eweill reply on @abagshaw comment.

[brianantonelli]

You can't just supply the pb, you have to freeze the graph which essentially merges your pb and checkpoint. More info: https://www.tensorflow.org/extend/tool_developers/#freezing

I have an issue open regarding freezing here: https://github.com/thtrieu/darkflow/issues/286

[jeffxtang]

@eweill have you got this resolved? I have the same problem and also tried the optimize_for_inference mentioned by @brianantonelli in https://github.com/thtrieu/darkflow/issues/286 but haven't been able to fix it...

@thtrieu thanks for the great repo. Can you please share your iOS code that successfully uses the model such as yolo.pb and tiny-yolo-voc.pb?

Thanks! Jeff

[jeffxtang]

I figured this out and just created a new iOS repo: https://github.com/jeffxtang/yolov2_tf_ios

[YanYas]

I'm working in Windows with the tensorflowsharp library and I'm coming across the same issue.

The first dimension of paddings must be the rank of inputs[4,2] [500,1000,3]
     [[Node: Pad = Pad[T=DT_FLOAT, Tpaddings=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_input_0_0, Pad/paddings)]]

@jeffxtang how did you get around this issue?

[apoorvkh]

Same issue using the TensorFlow library for Java/Android. @YanYas @jeffxtang -- do you have a solution?

[YanYas]

@apoorvkh sorry, not yet. Still looking into a work around.

[apoorvkh]

My team found a solution -- not sure how different the iOS implementation is, but we simply extended the expected input dimensions to 4. That is, for classifying a single 500x1000 RGB image, we gave the dimensions: [1,500,1000,3].