Understanding performance test

[yonatanbitton]

Hello. Thanks for the amazing repository!

I built the project and it works great.

I am trying to understand the performance test. I would be glad if you can elaborate about what was measured.

Looking at Measure.swift:

func 🎬🤚() {
        🏷(for: index, with: "end")

        let beforeMeasurement = getBeforeMeasurment(for: index)
        let currentMeasurement = measurements[index]
        if let startTime = currentMeasurement["start"],
            let endInferenceTime = currentMeasurement["endInference"],
            let endTime = currentMeasurement["end"],
            let beforeStartTime = beforeMeasurement["start"] {
            delegate?.updateMeasure(inferenceTime: endInferenceTime - startTime,
                                    executionTime: endTime - startTime,
                                    fps: Int(1/(startTime - beforeStartTime)))
        }

    }

startTime is the moment that the image is received. It is a pixelBuffer, (640x480) image. endInferenceTime is the moment when the inference was stopped, receiving a (14, 96, 96) heatmap from the CPM model. beforeStartTime is the start time of the previous frame. Why do we measure it?

If so I understand what is:

• inferenceTime: endInferenceTime - startTime - from pixelBuffer, (640x480) image to (14, 96, 96) heatmap .
• executionTime: endTime - startTime - from pixelBuffer, (640x480), including post-process (convertToPredictedPoints, and moving average filter. BTW - what is this filter?)

But I don't understand what is fps: Int(1/(startTime - beforeStartTime))) - why is the beforeStartTime relevant?

To summarize, my questions are:

1. Did I understand correctly the inferenceTime and executionTime?
2. Why is the fps calculated with beforeStartTime?
3. Why do you have moving average filter in the post-process code? Is it instead the gaussian_filter in the original CPM code?

Thank you very much.

[tucan9389]

@yonatanbitton

startTime is the moment that the image is received. It is a pixelBuffer, (640x480) image. endInferenceTime is the moment when the inference was stopped, receiving a (14, 96, 96) heatmap from the CPM model. beforeStartTime is the start time of the previous frame. Why do we measure it?

Exactly right

• inferenceTime: endInferenceTime - startTime - from pixelBuffer, (640x480) image to (14, 96, 96) heatmap .

Actually in our inferenceTime, it includes not only inference step but also pre-processing step because Apple's Vision framework is worked as the way.

• executionTime: endTime - startTime - from pixelBuffer, (640x480), including post-process (convertToPredictedPoints, and moving average filter. BTW - what is this filter?)

The output of CoreML inference is heatmaps (not points), so we should convert it into points. Including converting into points with inferenceTime is executionTime. So I think your explanation is right.

moving average filter. BTW - what is this filter?

This is a kind of smoothing method in signal processing. You can check the wiki(https://en.wikipedia.org/wiki/Moving_average if you want more detail!

So..

1. Did I understand correctly the inferenceTime and executionTime?

Yes

2. Why is the fps calculated with beforeStartTime?

For considering the camera video's fps. I set the fps of camera under 60. So the real fps shouldn't be under 60. And I wanted to check the real fps.

3. Why do you have moving average filter in the post-process code? Is it instead the gaussian_filter in the original CPM code?

Oh, you also saw the original code! At that time, I had three thinks.

1. The gaussian filter needs a lots of calculations in mobile device
2. I want to implement guassian filter without OpenCV but with Accelerate or Metal framework. But I was busy a little bit.
3. There is a low resolution issue in the model.

I hope you helpful.

[yonatanbitton]

Thank you for the answer :-)

Regarding the FPS:

I’d like to understand the relation between ‘Total Time’ and ‘FPS’ described in the tables here. Taking cpm/11-pro as an example, the total runtime is 23 msec (~43 fps) but the FPS for that setting is 15. You mentioned that you limited the camera to 60 FPS, so why don’t you get an FPS of 43 = min(43, 60) in this case? Is it because of background tasks running on the device, or a different reason?