VistanaGit commented 2 years ago

I have an issue about face recognition using FaceNet on RKNN. My main problem is how use converted model (I already converted pb model to rknn) embedding and recognition process. The steps that I did for this puprpose id as follows,

facenet orginal model: https://drive.google.com/open?id=1EXPBSXwTaqrSC0OhUdXNmKSh9qJUQ55-

Downloaded file from above link is: 20180402-114759.pb

python script to convert orginal model to rknn model: As described in : https://t-rock--chips-com.translate.goog/forum.php?mod=viewthread&tid=699&_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=nl&_x_tr_pto=nui

Our script for this conversion (inspired from the above link) is as follows:

import cv2 import numpy as np

from rknn.api import RKNN INPUT_SIZE = 160

if name == 'main':

# Create RKNN object
rknn = RKNN(verbose=True, verbose_file='./test1.log')

# Config for Model Input PreProcess
rknn.config(channel_mean_value='0 0 0 1', reorder_channel='0 1 2')
print('config done')

#load tensorflow model
print('--> Loading model')
rknn.load_tensorflow(tf_pb='./model/20180402-114759.pb',
                     inputs=['input'],
                     outputs=['embeddings'], 
                     #outputs=['InceptionResnetV1/Conv2d_1a_3x3/Relu'],
                     input_size_list=[[INPUT_SIZE, INPUT_SIZE, 3]])                  
print('done')

# Build Model
print('--> Building model')
rknn.build(do_quantization=False, dataset='./dataset.txt')
print('done')

# Export RKNN Model
rknn.export_rknn('./facenet_emd_out.rknn')

rknn.release()
------------------------------------------------------------------------

Run rknn model,

The problem is here:

After run the rknn model, logically, the output data from the orginal pb model (named as in own script) and Reshape_1.npy (named in own script) from rknn model ashould be same but they are different. But it's actually different. Output: pc_data.shape: (1, 512) u_data.shape: (1, 1, 512) For this difference, we reshped the rknn output to (1,512). After this reshape, the model is working file, but the distance matrix is quite far from expected reult and the recognition will be failed

Script for run model and face recognition is as follows:

Create RKNN object

rknn = RKNN(verbose=True, verbose_file='./test2.log')

# Load rknn model
print('--> Loading rknn model')
rknn.load_rknn('./facenet_emd_out.rknn')
print('done')

# init runtime environment
print('--> Init runtime environment')
ret = rknn.init_runtime(target='rk1808', device_id='a47426cd0f5f162b')
#ret = rknn.init_runtime(target='rk1808', target_sub_class='AICS')
if ret != 0:
    print('Init runtime environment failed')
    exit(ret)
print('done')

Inference

testtime=timer()
print('--> Running model') curImg_out_emd = rknn.inference(inputs=[img]) print('done')

curImg_out_emd_array= np.array(curImg_out_emd) curImg_out_emd_reshaped = curImg_out_emd_array.reshape(1,512) ###### THE RESHAPE PART THET WE DID print('curImg_out_emd shape: ' + str(curImg_out_emd_reshaped.shape))

Calculate distance matrix

print('Distace matrix:') for i in range(len(np_ds_list)):

print(' %1d ' % i, end='')

dist = np.sqrt(np.sum(np.square(np.subtract(np_ds_list[i][0,:], curImg_out_emd_reshaped[0])))) ## DISTANCE FOR RECOGNITION
#print(f'{str(i):<4}' + '      ' + f'{str(npyFileNames[i]):<25}' + '      '  + '% 1.4f' % dist)

if dist < 0.20:
    print(f'{str(i):<4}' + '      ' + f'{str(npyFileNames[i]):<25}' + '      '  + '% 1.4f' % dist)

if npyFileNames[i] == "User_Name":
   print('------------------------------------------------------------------------------------------')
   print(str(i) + '      ' + f'{str(npyFileNames[i]):<10}' + '      '  + '% 1.4f' % dist)                    
   print('------------------------------------------------------------------------------------------')

As we test the output from orginal pm model is working properly but the output from rknn is different, especially for larget dataset test.

Question: How set the parameter for model conversion? How change the output shape from rknn to be consistet with pb model ()with emebedding, Relu4 or any others) How measure the distance from rknn model out model from instances in dataset?

eRaul commented 2 years ago

When the model is converted, the mean_values and std_values (channel_mean_values is deprecated) set in the config interface should be the same as those during model training. If the RGB image is used for training, reorder_channel should be set to '0 1 2', otherwise set to '2 1 0'.
RKNN Toolkit will put all outputs to a list, so if you want to get first output, you can get it with outputs[0], the outputs[0]`s shape should be same with pb.
You can use the cosine distance to evaluate the difference between the results of the original model. For floating-point models, the cosine example is usually greater than 0.99 to be normal.

iwantofun commented 10 months ago