images are not saved to the local directory

[Athuliva]

in the readme it't mentioned that images will be saved to directory images before and after alignment will be stored in images. . Images are not saved into the directory (only the embedding is stored the local directory). Am i doing something wrong? . Also, can you please tell me how is the alignment done? does it uses keypoints?

Thank you

[zhouyuchong]

@Athuliva

• use gst-nvinfer-custom to save original/aligned surfaces local. I'm not sure which version you using, you can check the source code yourself.
• yes, it uses keypoints generated by previous detector and then use opencv to do cv::warpAffine. there are 5 lmks in retinaface, you can modify the code if you want to use other face detector.

[Athuliva]

do you have a standalone code to register a person; i.e to generate face of a person from an image and later compare with the embedding from the deepstream to identify the person in video file or rtsp stream.

[zhouyuchong]

@Athuliva yes but sorry i can't provide it. It's not difficult anyway, you can find many examples on github.

[Athuliva]

do i have to compile nvdsparse_retinaface.cpp?

[Athuliva]

@Athuliva yes but sorry i can't provide it. It's not difficult anyway, you can find many examples on github. i used the codes from insight face to aling and take the embeddings, but cosine similarity between the results from the deespstream and insightface are less than 0.7

assert lmk.shape == (5, 2)
assert image_size%112==0 or image_size%128==0
if image_size%112==0:
ratio = float(image_size)/112.0
diff_x = 0
else:
ratio = float(image_size)/128.0
diff_x = 8.0*ratio
dst = arcface_dst * ratio
dst[:,0] += diff_x
tform = trans.SimilarityTransform()
tform.estimate(lmk, dst)
M = tform.params[0:2, :]
return M

def norm_crop(img, landmark, image_size=112, mode='arcface'):
M = estimate_norm(landmark, image_size, mode)
warped = cv2.warpAffine(img, M, (image_size, image_size), borderValue=0.0)
return warped

def get(self, img, face):
aimg = face_align.norm_crop(img, landmark=face.kps, image_size=self.input_size[0])
face.embedding = self.get_feat(aimg).flatten()
return face.embedding

i used the engine file used in deepstream application in arcface-r100.cpp and compared with insightface recognition and ensured the engine file and onnx are giving same result without alignment.

[zhouyuchong]

@Athuliva you can save pictures from different stages by using this. However this repo needs a little change before use the newer gst-nvinfer-custom. If you are using the old version, you can check here

Again, deepstream gst-nvinfer uses the tensorrt plan file as original tensorrt does. If there are differences and you are sure the plan files are good, check inputs, in other words, check the preprocess methods. If the problem is with alignment stage, well, all source codes are here and you can test and modify them as you like :smile:

[Athuliva]

can you please tell me where should i place these parameters?

alignment-type=2
alignment-parent=2
alignment-pics=1
alignment-debug-level=3

inside the config file of retinaface?

[zhouyuchong]

@Athuliva I think they should be in arcface's config file in the old version.

[Athuliva]

without alignment also the results are not same this is the config file of arcface

[property]
gpu-id=0
gie-unique-id=2
model-engine-file=arcface_63/arcface-r100.engine
#model-engine-file=/opt/models/arcface/arcface_new.engine
# batch-size=1
net-scale-factor=0.0078125
offsets=127.5;127.5;127.5
force-implicit-batch-dim=1
model-color-format=0
network-mode=2
process-mode=2
network-type=100
output-tensor-meta=1
symmetric-padding=0
classifier-async-mode=0
operate-on-gie-id=1
operate-on-class-ids=0
input-object-min-width=10
input-object-min-height=10
infer-dims=3;112;112
uff-input-order=0
maintain-aspect-ratio=0

user-meta=1

modified the arcface-r100.cpp in tensortx repo (added a resize function)

    std::string folderPath = "../test_images";
    DIR* dir = opendir(folderPath.c_str());
    if (dir == nullptr) {
        std::cerr << "Could not open directory: " << folderPath << std::endl;
        return 0;
    }

    struct dirent* entry;
    while ((entry = readdir(dir)) != nullptr) {
        std::string filename = entry->d_name;
        if (filename == "." || filename == "..") {
            continue;
        }
        std::string imagePath = folderPath + "/" + filename;
        struct stat pathStat;
        stat(imagePath.c_str(), &pathStat);
        if (S_ISREG(pathStat.st_mode) && imagePath.substr(imagePath.find_last_of(".") + 1) == "jpg") {

            std::string newPath = changeFilePath(imagePath);
            cv::Mat img = cv::imread(imagePath);
            cv::resize(img, img, cv::Size(112, 112));
            for (int i = 0; i < INPUT_H * INPUT_W; i++) {
        data[i] = ((float)img.at<cv::Vec3b>(i)[2] - 127.5) * 0.0078125;
        data[i + INPUT_H * INPUT_W] = ((float)img.at<cv::Vec3b>(i)[1] - 127.5) * 0.0078125;
        data[i + 2 * INPUT_H * INPUT_W] = ((float)img.at<cv::Vec3b>(i)[0] - 127.5) * 0.0078125;

    }

    // Run inference
    auto start = std::chrono::system_clock::now();
    doInference(*context, data, prob, BATCH_SIZE);
    auto end = std::chrono::system_clock::now();
    std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(end - start).count() << "ms" << std::endl;

    cv::Mat out(512, 1, CV_32FC1, prob);
    cv::Mat out_norm;
    cv::normalize(out, out_norm);

    writeToFile(newPath, out);
         }
    }

similarity between the results from the modified arcface-r100.cpp and deepstream output

[zhouyuchong]

@Athuliva I validate the result and it's perfect. Maybe there is still something missing in your codes. I will organize and update my codes once I have time. Here is the config file that works for me.

[property]
gpu-id=0
gie-unique-id=2
model-engine-file=../../models/arcface/arcface-r50.engine
# batch-size=1
net-scale-factor=0.0078125
offsets=127.5;127.5;127.5
force-implicit-batch-dim=1
model-color-format=1
network-mode=2
process-mode=2
network-type=100
output-tensor-meta=1
symmetric-padding=1
classifier-async-mode=0
# output-blob-names=prob
operate-on-gie-id=1
operate-on-class-ids=0
#input-object-min-width=50
#input-object-min-height=50

alignment-type=1
alignment-pics-path=/opt/nvidia/deepstream/deepstream-6.3/sources/video_analysis_platform/images/face/

[Athuliva]

is this with or without alignment?

[zhouyuchong]

@Athuliva it's with alignment

[Athuliva]

arcface-r100.cpp code expects RGB input, so the model-color-format=0, right?

[zhouyuchong]

@Athuliva Yes you're right, you can have a try.

[Athuliva]

i guess the converter is already changing the image to RGBA

[zhouyuchong]

@Athuliva I've updated this repo, now it is more simple and more "demo", the outputs are now visualized. have a try :smile:

[Athuliva]

@zhouyuchong its easy and simple now!! thank you