Pre-trained model on Android application

[MagedSaeed]

Hi

First of all, thanks for this work. My question is that can I use the pre-trained weights you provide in an android application? if so, how can I create the .pb file for this purpose?

Thanks in advance,,

[yu4u]

This project is released under the MIT license but the pretrained model (or the training dataset) is restricted by the IMDB-WIKI dataset's license. Therefore, please check the dataset website.

https://data.vision.ee.ethz.ch/cvl/rrothe/imdb-wiki/

Please notice that this dataset is made available for academic research purpose only. All the images are collected from the Internet, and the copyright belongs to the original owners. If any of the images belongs to you and you would like it removed, please kindly inform us, we will remove it from our dataset immediately.

Please refer to the following link for export (but I did NOT try it).

https://stackoverflow.com/questions/37704362/tensorflow-freeze-graph-script-failing-on-model-defined-with-keras

[MagedSaeed]

I am doing this for the purpose of my senior project in my university. If we will publish it on the store, then we will study those licenses deeply. Regarding the model, I figured out a way to do so. The code is taken from the following gist: https://gist.github.com/fsausset/57b99a3db5e1a05569845894ec385eef with very minor modifications. The code is below:

from keras.models import Sequential
from keras.models import model_from_json
from keras import backend as K
from wide_resnet import WideResNet
import tensorflow as tf
from tensorflow.python.tools import freeze_graph
import os

# Load existing model.
#with open("model.json",'r') as f:
 #   modelJSON = f.read()

#model = model_from_json(modelJSON)

model = WideResNet(64, 16, 8)()
model.load_weights("weights.18-4.06.hdf5")

# All new operations will be in test mode from now on.
K.set_learning_phase(0)

# Serialize the model and get its weights, for quick re-building.
config = model.get_config()
print(model.summary())
print(config)
weights = model.get_weights()

# Re-build a model where the learning phase is now hard-coded to 0.
new_model = model.from_config(config)
new_model.set_weights(weights)

temp_dir = "graph"
checkpoint_prefix = os.path.join(temp_dir, "saved_checkpoint")
checkpoint_state_name = "checkpoint_state"
input_graph_name = "input_graph.pb"
output_graph_name = "output_graph.pb"

# Temporary save graph to disk without weights included.
saver = tf.train.Saver()
checkpoint_path = saver.save(K.get_session(), checkpoint_prefix, global_step=0, latest_filename=checkpoint_state_name)
tf.train.write_graph(K.get_session().graph, temp_dir, input_graph_name)

input_graph_path = os.path.join(temp_dir, input_graph_name)
input_saver_def_path = ""
input_binary = False
output_node_names = "dense_1/Softmax,dense_2/Softmax" # model dependent
restore_op_name = "save/restore_all"
filename_tensor_name = "save/Const:0"
output_graph_path = os.path.join(temp_dir, output_graph_name)
clear_devices = False

# Embed weights inside the graph and save to disk.
freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
                          input_binary, checkpoint_path,
                          output_node_names, restore_op_name,
                          filename_tensor_name, output_graph_path,
                          clear_devices, "")

To make sure of the code reliability, These are the output nodes names of the model: dense_1/Softmax,dense_2/Softmax. one is for the gender and the other is for the age. I hope that my other parameters are consistent with the pretrained weights and I am not doing something wrong.

I have been successful in generating the .pb file.I put it in my Android App. I am detecting faces using google vision API and on my way to crop them to 64643 to be suitable for the model. I have tested, for the purpose of trying and fun, the model on the whole image "NOT ONLY THE CROPPED FACE" and it give me 0.5 prediction for gender which, I guess, is fine. :)

We may re-train the model to image size of 227 for better performance. Any feedback is highly appreciated. Thanks in advance.

[MagedSaeed]

Hi One more remark, I found that the gender detection on Android is a little bit weird. Results for me are always arount 50% [random guess]. However, it seems that it did a better job on the age classification where the highest value is reasonable. What am i doing wrong?

[yu4u]

I found that the gender detection on Android is a little bit weird. Results for me are always arount 50% [random guess]. However, it seems that it did a better job on the age classification where the highest value is reasonable.

This is very wired. It is more likely that both predictions becomes random guesses.

Several things may cause problems:

• Channel order: the channel order of the input should be BGR, not RGB.
• Margin of face region: there is 40% padding on all sides (top, bottom, left, right) for a face region obtained by face detector.

(But these probably are not a problem here...)

[MagedSaeed]

I found my way to solve the problem, but your answer is not making sense for me.

First of all, The problem of random guessing was both on the age and gender, but for age it was not that obvious as it gives high values for predicted age but not discriminating. for example, it gives for age 2 a probability of 0.001 and for the 24, which is a good prediction, a value of 0.007.

I realized that this is a problem with the image input, I revised the work and found that Bitmap.getPixels method I am using to get pixel values of the image is reversing color channels from RGB to BGR. I just reversed the order and it works for me.

What is wired in your response here is that you are saying :

Channel order: the channel order of the input should be BGR, not RGB.

However, it works the other way around for me!!

Moreover, in the demo.py you are converting the color channels to RGB as well. This line I am taking it from the file:

input_img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

I also noticed that there a small difference between the values of demo.py and my .pb predictions on android app. I did not margin the face region and that might be one reason.

Here is my answer on the stack summarizing my work, https://stackoverflow.com/a/49582776/4412324

Thanks for your response @yu4u I really appreciate.

[yu4u]

Congrats @MagedSaeed!

Moreover, in the demo.py you are converting the color channels to RGB as well.

input_img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

This is only used as the input to the dlib detector,

detected = detector(input_img, 1)

and for the model prediction, the original image is used:

faces[i, :, :, :] = cv2.resize(img[yw1:yw2 + 1, xw1:xw2 + 1, :], (img_size, img_size))
...
results = model.predict(faces)

[MagedSaeed]

Thanks @yu4u

Regarding BGR2RGB, you are right and I am the one who got confused. I converted the image to BGR instead of RGB in the app but thought the other way around.

I really appreciate your help. Thanks,

I am going to close this issue as it has been already resolved.

I have a suggestion regarding the face detector, Why not to use google vision API instead of dlib? Although dlib gives better results with face landmarks, it performs badly on non-frontal faces. I just input a static image to the demo.py where the face is slightly rotated and the detector did not detect the face!!

Google vision API is much more better in this point. It can detect non-frontal faces with very good accuracy. I have it in my app and was a good job, better than dlib. You may watch this official YouTube video if you are interested. https://www.youtube.com/watch?v=qu1RjE_2zhQ

Best wishes and Regards,,, :100:

[yu4u]

Although dlib gives better results with face landmarks, it performs badly on non-frontal faces.

Thank you for the suggestion. Yes, the frontal_face_detector used in the demo is not good. However, latest dlib supports dlib.cnn_face_detection_model_v1 detector, which can detect non-frontal faces (at the cost of additional computational cost). I would update my code to use the dlib.cnn_face_detection_model_v1.

[ruobop]

Hello @MagedSaeed , Can I ask about inference time on Android device? I tested on an RK3288 device and got inference time around 1800ms (measure only sess.run cost time), which is pretty slow. Did you use any optimization method like model quantization?

[MagedSaeed]

Hi @ruobop

I agree that it is pretty slow inference. I do not know any of these methods you talked about. Could you give me explanation, Please.