Run ssd w/ mobilenet using tensorflow C++ API is slower than tensorflow python API

[kikirizki]

Recently I try to run ssd with mobilenet model from tensorflow object detection api. I try to run realtime object detection using tensorflow+opencv on both python and c++, surprisingly the c++ is much slower than the python API. I am sure that I run both of them with GPU support, here is my code

/* Copyright 2015 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/

// A minimal but useful C++ example showing how to load an Imagenet-style object
// recognition TensorFlow model, prepare input images for it, run them through
// the graph, and interpret the results.
//
// It's designed to have as few dependencies and be as clear as possible, so
// it's more verbose than it could be in production code. In particular, using
// auto for the types of a lot of the returned values from TensorFlow calls can
// remove a lot of boilerplate, but I find the explicit types useful in sample
// code to make it simple to look up the classes involved.
//
// To use it, compile and then run in a working directory with the
// learning/brain/tutorials/label_image/data/ folder below it, and you should
// see the top five labels for the example Lena image output. You can then
// customize it to use your own models or images by changing the file names at
// the top of the main() function.
//
// The googlenet_graph.pb file included by default is created from Inception.
//
// Note that, for GIF inputs, to reuse existing code, only single-frame ones
// are supported.

#include <fstream>
#include <utility>
#include <vector>
#include <iostream>
#include <opencv2/core/mat.hpp>
#include <opencv2/videoio.hpp>
#include <opencv/cv.hpp>

#include "tensorflow/cc/ops/const_op.h"
#include "tensorflow/cc/ops/image_ops.h"
#include "tensorflow/cc/ops/standard_ops.h"
#include "tensorflow/core/framework/graph.pb.h"
#include "tensorflow/core/framework/tensor.h"
#include "tensorflow/core/graph/default_device.h"
#include "tensorflow/core/graph/graph_def_builder.h"
#include "tensorflow/core/lib/core/errors.h"
#include "tensorflow/core/lib/core/stringpiece.h"
#include "tensorflow/core/lib/core/threadpool.h"
#include "tensorflow/core/lib/io/path.h"
#include "tensorflow/core/lib/strings/stringprintf.h"
#include "tensorflow/core/platform/env.h"
#include "tensorflow/core/platform/init_main.h"
#include "tensorflow/core/platform/logging.h"
#include "tensorflow/core/platform/types.h"
#include "tensorflow/core/public/session.h"
#include "tensorflow/core/util/command_line_flags.h"

// These are all common classes it's handy to reference with no namespace.
using tensorflow::Flag;
using tensorflow::Tensor;
using tensorflow::Status;
using tensorflow::string;
using tensorflow::int32;

using namespace std;

// Takes a file name, and loads a list of labels from it, one per line, and
// returns a vector of the strings. It pads with empty strings so the length
// of the result is a multiple of 16, because our model expects that.
Status ReadLabelsFile(const string& file_name, std::vector<string>* result,
                      size_t* found_label_count) {
    std::ifstream file(file_name);
    if (!file) {
        return tensorflow::errors::NotFound("Labels file ", file_name,
                                            " not found.");
    }
    result->clear();
    string line;
    while (std::getline(file, line)) {
        result->push_back(line);
    }
    *found_label_count = result->size();
    const int padding = 16;
    while (result->size() % padding) {
        result->emplace_back();
    }
    return Status::OK();
}

static Status ReadEntireFile(tensorflow::Env* env, const string& filename,
                             Tensor* output) {
    tensorflow::uint64 file_size = 0;
    TF_RETURN_IF_ERROR(env->GetFileSize(filename, &file_size));

    string contents;
    contents.resize(file_size);

    std::unique_ptr<tensorflow::RandomAccessFile> file;
    TF_RETURN_IF_ERROR(env->NewRandomAccessFile(filename, &file));

    tensorflow::StringPiece data;
    TF_RETURN_IF_ERROR(file->Read(0, file_size, &data, &(contents)[0]));
    if (data.size() != file_size) {
        return tensorflow::errors::DataLoss("Truncated read of '", filename,
                                            "' expected ", file_size, " got ",
                                            data.size());
    }
    output->scalar<string>()() = data.ToString();
    return Status::OK();
}

// Given an image file name, read in the data, try to decode it as an image,
// resize it to the requested size, and then scale the values as desired.
Status ReadTensorFromImageFile(const string& file_name, const int input_height,
                               const int input_width, const float input_mean,
                               const float input_std,
                               std::vector<Tensor>* out_tensors) {
    auto root = tensorflow::Scope::NewRootScope();
    using namespace ::tensorflow::ops;  // NOLINT(build/namespaces)

    string input_name = "file_reader";
    string output_name = "normalized";

    // read file_name into a tensor named input
    Tensor input(tensorflow::DT_STRING, tensorflow::TensorShape());
    TF_RETURN_IF_ERROR(
            ReadEntireFile(tensorflow::Env::Default(), file_name, &input));

    // use a placeholder to read input data
    auto file_reader =
            Placeholder(root.WithOpName("input"), tensorflow::DataType::DT_STRING);

    std::vector<std::pair<string, tensorflow::Tensor>> inputs = {
            {"input", input},
    };

    // Now try to figure out what kind of file it is and decode it.
    const int wanted_channels = 3;
    tensorflow::Output image_reader;
    if (tensorflow::StringPiece(file_name).ends_with(".png")) {
        image_reader = DecodePng(root.WithOpName("png_reader"), file_reader,
                                 DecodePng::Channels(wanted_channels));
    } else if (tensorflow::StringPiece(file_name).ends_with(".gif")) {
        // gif decoder returns 4-D tensor, remove the first dim
        image_reader =
                Squeeze(root.WithOpName("squeeze_first_dim"),
                        DecodeGif(root.WithOpName("gif_reader"), file_reader));
    } else {
        // Assume if it's neither a PNG nor a GIF then it must be a JPEG.
        image_reader = DecodeJpeg(root.WithOpName("jpeg_reader"), file_reader,
                                  DecodeJpeg::Channels(wanted_channels));
    }
    // Now cast the image data to float so we can do normal math on it.
    // auto float_caster =
    //     Cast(root.WithOpName("float_caster"), image_reader, tensorflow::DT_FLOAT);

    auto uint8_caster =  Cast(root.WithOpName("uint8_caster"), image_reader, tensorflow::DT_UINT8);

    // The convention for image ops in TensorFlow is that all images are expected
    // to be in batches, so that they're four-dimensional arrays with indices of
    // [batch, height, width, channel]. Because we only have a single image, we
    // have to add a batch dimension of 1 to the start with ExpandDims().
    auto dims_expander = ExpandDims(root.WithOpName("dim"), uint8_caster, 0);

    // Bilinearly resize the image to fit the required dimensions.
    // auto resized = ResizeBilinear(
    //     root, dims_expander,
    //     Const(root.WithOpName("size"), {input_height, input_width}));

    // Subtract the mean and divide by the scale.
    // auto div =  Div(root.WithOpName(output_name), Sub(root, dims_expander, {input_mean}),
    //     {input_std});

    //cast to int
    //auto uint8_caster =  Cast(root.WithOpName("uint8_caster"), div, tensorflow::DT_UINT8);

    // This runs the GraphDef network definition that we've just constructed, and
    // returns the results in the output tensor.
    tensorflow::GraphDef graph;
    TF_RETURN_IF_ERROR(root.ToGraphDef(&graph));

    std::unique_ptr<tensorflow::Session> session(
            tensorflow::NewSession(tensorflow::SessionOptions()));
    TF_RETURN_IF_ERROR(session->Create(graph));
    TF_RETURN_IF_ERROR(session->Run({inputs}, {"dim"}, {}, out_tensors));
    return Status::OK();
}

// Reads a model graph definition from disk, and creates a session object you
// can use to run it.
Status LoadGraph(const string& graph_file_name,
                 std::unique_ptr<tensorflow::Session>* session) {
    tensorflow::GraphDef graph_def;
    Status load_graph_status =
            ReadBinaryProto(tensorflow::Env::Default(), graph_file_name, &graph_def);
    if (!load_graph_status.ok()) {
        return tensorflow::errors::NotFound("Failed to load compute graph at '",
                                            graph_file_name, "'");
    }
    session->reset(tensorflow::NewSession(tensorflow::SessionOptions()));
    Status session_create_status = (*session)->Create(graph_def);
    if (!session_create_status.ok()) {
        return session_create_status;
    }
    return Status::OK();
}

int main(int argc, char* argv[]) {
    // These are the command-line flags the program can understand.
    // They define where the graph and input data is located, and what kind of
    // input the model expects. If you train your own model, or use something
    // other than inception_v3, then you'll need to update these.
    string image(argv[1]);
    string graph ="ssd_mobilenet_v1_coco_11_06_2017/frozen_inference_graph.pb";
    string labels ="mscoco_label_map.pbtxt";
    int32 input_width = 299;
    int32 input_height = 299;
    int32 depth =3;
    float input_mean = 0;
    float input_std = 255;
    string input_layer = "image_tensor:0";
    vector<string> output_layer ={ "detection_boxes:0", "detection_scores:0", "detection_classes:0", "num_detections:0" };

    bool self_test = false;
    string root_dir = "";
  //============================================//
    cv::VideoCapture cap(0);
    // creating a Tensor for storing the data
    tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1,input_height,input_width,depth}));
    auto input_tensor_mapped = input_tensor.tensor<float, 4>();
    cv::Mat img_stream;
    const float * source_data;
    // copying the data into the corresponding tensor
    cv::namedWindow("camera");
    cv::Size s(input_height,input_width);
    cv::Mat resized_img;
    cv::Mat resized_i;
    std::unique_ptr<tensorflow::Session> session;
    string graph_path = tensorflow::io::JoinPath(root_dir, graph);
    std::vector<Tensor> resized_tensors;
    string image_path = tensorflow::io::JoinPath(root_dir, image);
    Status read_tensor_status;
    std::vector<Tensor> outputs;
    int image_width;
    int image_height;

    // First we load and initialize the model.
    LoadGraph(graph_path, &session);
    ReadTensorFromImageFile(image_path, input_height, input_width, input_mean,
                            input_std, &resized_tensors);

    for(;;) {
        cap>>img_stream;
//        cv::resize(img_stream,resized_i,s,0,0,cv::INTER_CUBIC);
//        resized_i.convertTo(resized_img, CV_32FC1);
//        source_data = (float*) resized_img.data;
//        for (int y = 0; y < input_height; ++y) {
//            const float *source_row = source_data + (y * input_width * depth);
//            for (int x = 0; x < input_width; ++x) {
//                const float *source_pixel = source_row + (x * depth);
//                for (int c = 0; c < depth; ++c) {
//                    const float *source_value = source_pixel + c;
//                    input_tensor_mapped(0, y, x, c) = *source_value;
//                }
//            }
//        }
        //============================================//
        const Tensor& resized_tensor = resized_tensors[0];
        // << ",data:" << resized_tensor.flat<tensorflow::uint8>();
        // Actually run the image through the model.

        session->Run({{input_layer, resized_tensor}},
                                         output_layer, {}, &outputs);
        image_width = resized_tensor.dims();
        image_height = 0;
        //int image_height = resized_tensor.shape()[1];
        //tensorflow::TTypes<float>::Flat iNum = outputs[0].flat<float>();
        tensorflow::TTypes<float>::Flat scores = outputs[1].flat<float>();
        tensorflow::TTypes<float>::Flat classes = outputs[2].flat<float>();
        tensorflow::TTypes<float>::Flat num_detections = outputs[3].flat<float>();
        auto boxes = outputs[0].flat_outer_dims<float,3>();
//        LOG(ERROR) << "num_detections:" << num_detections(0) << "," << outputs[0].shape().DebugString();
        for(size_t i = 0; i < num_detections(0) && i < 20;++i)
        {
            if(scores(i) > 0.5)
            {
                cout<< i << ",score:" << scores(i) << ",class:" << classes(i)<< ",box:" << "," << boxes(0,i,0) << "," << boxes(0,i,1) << "," << boxes(0,i,2)<< "," << boxes(0,i,3);
            }
        }
        cv::imshow("camera",img_stream);
        if (cv::waitKey(33) == 27)
            break;
    }

    return 0;
}

Note : Since I do not know how to convert cv::Mat into tensor, I do not actually detect the object from the camera for each frame instead I substitute the cv::Mat with image from image file ("cat.jpg"). My intention is to compare the performance between tensorflow C++ and python API for object detection w mobilenet.. thanks,

oke then why the C++ API is slower? thanks

[memo]

Hi, thanks for the question. I'm not sure why this code is much slower than your python code (since I can't see the python code), but I highly doubt that tensorflow itself is slower in C++. If you want to compare the tensorflow C++ API performance to the python API performance, I'd suggest remove everything from your update loop that isn't directly related to tensorflow. I.e. get rid of the video capture and scoring, and only leave the session->run. Then gradually introduce other elements.

By the way, apologies if this is a silly question, but you do know that cv::waitKey(33) means 'sleep for 33 milliseconds' right? And you do the same in python?

[kikirizki]

I am sorry my bad. I didn't aware that I write waitKey(33), I just copy and paste from my old webcam program, oh silly me. Now it is work and a little faster than the python counterpart. I think the problem is solved, thank you

[memo]

Ok great to hear that :)

[kerolos]

hello @kikirizki @memo , could you please explain to me how can I run the above code to my Linux machine? i have already built Tensorflow by using bazal builder. but I do not know what is the following steps to compile and run your prediction code using SSD MODEL. I will really appreciate your advice. and thanks in advance

[kikirizki]

hello @kerolos ... sorry for late reply, just try to clone my repo https://github.com/memo/ofxMSATensorFlow/issues/34#issuecomment-366619818

I have compile it my self and there is a script in the repo to run it, hope its help

[kikirizki]

here is the repo https://github.com/memo/ofxMSATensorFlow/issues/34#issuecomment-366619818

[kikirizki]

i mean here is the repo https://github.com/kikirizki/tensorflow-nobazel

[kerolos]

thanks a lot @kikirizki for your reply and help :) .

[abhigoku10]

@kikirizki @memo hi guys i am also facing the same issue my python code takes a time of 4 sec and my c ++ code takes 18 secs , i have frozen.pb model which i use to create the binary can you give me some suggestions on it