ic01 test case can not produce accuracy stat in performance multi mode

[ninn55]

Solved

Problem Description

In the image classification test case. At the end of multiple runs (to test accuracy), runner software says:

01161.646 parser: Post processing benchmark run...
01161.653 ulp-mlperf: Not all of the inputs had an output result

How to reproduce this:

First flash the image classification firmware to MCU. Then run the runner software. After detecting the serial connection, choose ML Performance 1.0.0, Then click initialize. Then choose Multiple and Run. After all test cases are inferenced and their result returned. The runner software says.

01161.646 parser: Post processing benchmark run...
01161.653 ulp-mlperf: Not all of the inputs had an output result

My system info:

The runner software is running on ubuntu 18.04. Runner software version 3.0.3 (It is the latest version as far as I can tell).

Extras

As far as I can tell, this problem is caused by some .bin files that do not have their label in y_labels.csv.

There are 200 labels from the CSV

file

root@NUC:~/eembc/runner/benchmarks/ulp-mlperf/datasets# cat ./ic01/y_labels.csv | wc -l
200

But 249 .bin test cases

root@NUC:~/eembc/runner/benchmarks/ulp-mlperf/datasets# find ./ic01 -type f -name *.bin | wc -l
249

Clearly, there are not enough labels for everyone.

So I wrote this cute script to get rid of some excess .bin files.

from os import listdir, remove
from pandas import read_csv

data_files = listdir("ic01")
label_files = list(read_csv("ic01/y_labels.csv", header=None, usecols=[0])[0])

# Remove not labeled file
for f in data_files:
    if f.endswith(".csv"):
        continue
    if (f not in label_files) and f.endswith(".bin"):
        remove("./ic01/" + f)

data_files = listdir("ic01")
label_files = list(read_csv("ic01/y_labels.csv", header=None, usecols=[0])[0])

# Reverse double check
for i in label_files:
    if i not in data_files and i.endswith(".bin"):
        raise RuntimeError

But SADLY after deleting .bin files with no label, the runner software still says Not all of the inputs had an output result.

This zip file is my whole session files. 20210419152722.zip

Am I doing something wrong? Please Help.

EDIT:

solution

Turns out in submitter_implemented.cpp file I write th_result function wrong. So the problem is the inference result returned by serial to the runner software does not match the protocol it requires. But Not all of the inputs had an output result is kind of misleading in my POV.

Thanks

[petertorelli]

Hello @ninn55,

Thank you for filing such a thorough description. The runner will only use the files listed in the y_labels.csv ground-truth, but good debug!

The problem is the reference code for ic01. In the log file I see this:

00012.072 dut: m-warmup-start-0
00012.073 dut: m-warmup-done
00012.074 dut: m-infer-start-1
00012.076 dut: m-lap-us-559423445
00013.315 dut: m-lap-us-560664710
00013.316 dut: m-infer-done
00013.318 dut: m-results-[0
00013.318 dut: 0.000,0
00013.319 dut: 0.000,0
00013.320 dut: 0.000,0
00013.321 dut: 0.000,0
00013.322 dut: 0.000,0
00013.322 dut: 0.000,0
00013.323 dut: 0.000,99
00013.324 dut: 0.996,0
00013.325 dut: 0.000,0
00013.326 dut: 0.000]
00013.326 dut: m-ready

This is the wrong output format, it should be one line, e.g. m-results-[r0,r1,...,rn]. Looking at the code in the MLCommons/tiny repo, it looks like this has been fixed recently, see this issue.

Try pulling the latest code for image classification, it should fix the problem.

[ninn55]

Thank you. That's exactly the problem.