Performance test - slow with FaceEncodings

[unruledboy]

I did a step by step performance test using the 32bit.png in TestImages, on my Surface Book 2 i7 (i7-8650U quad-core + 16GB), it took about 0.712s in total:

• LoadImage: 0.03
• FaceEncodings: 0.71
• CompareFace: 0.03

I was wondering if we can improve the performance?

Here is the sample code:

    static void Main(string[] args)
        {
            var testImageDirectory = @"PATH_TO_TEST_FACES";
            var faceDetection = new FaceDetectionService(@"PATH_TO_MODELS");
            var trainedData = faceDetection.Train(new[]
            {
                new TrainFace{ Name = "Obama", ImageData = GetContent(testImageDirectory + "obama.jpg") },
                new TrainFace{ Name = "Obama", ImageData = GetContent(testImageDirectory + "obama2.jpg") },
                new TrainFace{ Name = "Obama", ImageData = GetContent(testImageDirectory + "obama3.jpg") },
            });
            faceDetection.Initialize(trainedData);
            Test(testImageDirectory + "32bit.png", faceDetection);
            Console.ReadLine();
        }

    private static void Test(string file, FaceDetectionService faceDetection)
        {
            var s = Stopwatch.StartNew();
            var (result, name) = faceDetection.RecognizeFast(file);
            s.Stop();
            Console.WriteLine($"{s.Elapsed}: {result} = {name} @ {file}");
        }

        static byte[] GetContent(string file) => File.ReadAllBytes(file);

    class NamedFaceEncodingData
    {
        public string Name { get; set; }
        public byte[] Encodings { get; set; }
    }

    class NamedFaceEncoding
    {
        public string Name { get; set; }
        public FaceEncoding Encoding { get; set; }
    }

    class FaceDetectionService
    {
        private readonly FaceRecognition faceRecognition;
        private IEnumerable<NamedFaceEncodingData> faceData;
        private IEnumerable<NamedFaceEncoding> faces;

        public FaceDetectionService(string modelPath)
        {
            faceRecognition = FaceRecognition.Create(modelPath);
        }

        public void Initialize(IEnumerable<NamedFaceEncodingData> faceData)
        {
            this.faceData = faceData;

            var bf = new BinaryFormatter();
            faces = faceData.Select(x =>
            {
                using var buffer = new MemoryStream(x.Encodings);
                var encoding = bf.Deserialize(buffer) as FaceEncoding;
                return new NamedFaceEncoding { Name = x.Name, Encoding = encoding };
            });
        }

        public (bool result, string name) Recognize(byte[] imageContent)
        {
            using var image = FaceRecognition.LoadImage((Bitmap)System.Drawing.Image.FromStream(new MemoryStream(imageContent)));
            var encodings = faceRecognition.FaceEncodings(image);
            foreach (var encoding in encodings)
            {
                var firstMatchedFace = faces.FirstOrDefault(x => FaceRecognition.CompareFace(x.Encoding, encoding));
                encoding.Dispose();
                if (firstMatchedFace != null)
                    return (true, firstMatchedFace.Name);
            }
            return (false, null);
        }

        public (bool result, string name) RecognizeFast(string file)
        {
            var s = Stopwatch.StartNew();
            using var image = FaceRecognition.LoadImageFile(file);
            Console.WriteLine($"LoadImage: {s.Elapsed}");

            var encodings = faceRecognition.FaceEncodings(image);
            Console.WriteLine($"Encodings: {s.Elapsed}");
            foreach (var encoding in encodings)
            {
                var firstMatchedFace = faces.FirstOrDefault(x => FaceRecognition.CompareFace(x.Encoding, encoding));
                encoding.Dispose();
                Console.WriteLine($"Compare: {s.Elapsed}");
                if (firstMatchedFace != null)
                    return (true, firstMatchedFace.Name);
            }
            return (false, null);
        }
    }

[takuya-takeuchi]

@unruledboy You tried FRDN cpu? You could improve performance by using FRDN.MKL. And reference of performance for MKL https://github.com/takuya-takeuchi/FaceRecognitionDotNet/issues/4#issuecomment-421370043

[unruledboy]

@takuya-takeuchi tried the MKL 2020 release as 2019 initial release couldn't be found. and libiomp5md.dll is also missing. that could have explained why it does not work (no perf difference) .

I noticed your code change in https://github.com/takuya-takeuchi/DlibDotNet/commit/44ae718d22d3a5324aaef4ddc5d94bb706edb83b , so I copied all dlls in the both bin / \runtimes\win-x64\native, no luck here. Not sure what I have missed.

And I agree we should bundle those dlls in the package.

[unruledboy]

Another way to improve the performance, maybe we can reduce the number of landmarks?

[takuya-takeuchi]

@unruledboy

Another way to improve the performance, maybe we can reduce the number of landmarks?

no. it is not slow to detect landmarks. What is the most performance issue is detecting face detection. But this depends on dlib method. We have no idea to control.

But we may improve face detection performance by using another library.

• Face Detection (Another Lib)
• Face Encoding (Pass the bounding box from Another lib and do FRDN)
• Face Verify (FRDN)