Handle TFLite backend description in case of delegate fallback

[anhappdev]

Per @d-uzlov's comment:

I should note that tflite backend uses strategy "use CPU if specified delegate failed", and we use iOS settings for Windows, so now we can see CoreML labels in the benchmark list in Windows builds. It's not the fault of this PR, but maybe we should do something about it in the future. Probably, create an issue to create a separate config for Windows.

Possible solution: A backend_description string is passed to the backend in RunSettings and receive it back in RunResult. The TFLite backend should modify backend_description in case of delegate fallback.

[anhappdev]

For now, I just display the backend name without the accelerator name, because of the following reasons:

1. To check which accelerator is actually used (with fallback handling) I need to modify the backend code. I want to avoid that, since the backend was well tested.
2. Sometimes TFLite can use multiples accelerators for the same model. So, it is not correct to just display one accelerator.

Therefore, weighting the time effort / side effect vs. benefits of this change, I choose to just display the backend name. This should be the same as the old Android app. If there is a request to display accelerators name, I will add it later.

Do we need to display the actually used accelerator name?

[relja128]

We should display the accelerator name, since whether or not we're using CPU / GPU / NPU really matters. Let's discuss next week when everyone is around.

[freedomtan]

@Mostelk and @mohitmundhragithub please comment on if it's fine to have accelerator name displayed in QTI and Samsung' backends.

[jwookiehong]

Freedom. I got you.

@mohitmundhragithub , @Mostelk , @shabanim to investigate if providing accelerator names(DSP, GPU, CPU, NPU or any combination thereof) is possible via vendor SDKs.

[shabanim]

I am confused a bit! It seems RunSettings and RunResults hold test settings. Some of these settings come from the UI such as mode. Some others are vendors specific parameters such as batch size. If my understanding is correct, device (i.e., CPU, GPU, etc.,) should be one of the parameters specified for testing.

Thus it should be specified in the generated report as well. Does this make sense to you?

[freedomtan]

I am confused a bit! It seems RunSettings and RunResults hold test settings. Some of these settings come from the UI such as mode. Some others are vendors specific parameters such as batch size. If my understanding is correct, device (i.e., CPU, GPU, etc.,) should be one of the parameters specified for testing.

Thus it should be specified in the generated report as well. Does this make sense to you?

In principle, benchmark settings should be from backend settings. Yes, there is a batch size hack to set it from UI from iOS devices (but that should always be considered as a hacks, in my opinion.

[freedomtan]

Use TfLite as example, we can report either delegates (NNAPI, GPU, (MediaTek's) Neuron, etc.) Or we want to show, e.g. Neuron MTK DLA or Neuron MTK DSP.

For Q, we can simple report (SNPE), or SNPE DSP (DSP+HTP), SNPE GPU, etc.

For Intel, OpenVINO, or (OpenVINO CPU, OpenVIO GPU, etc.)

Let's create a backend function pointer for each backend to report back the underling hardware (e.g., OpenVINO CPU, OpenVINO GPU, etc.)

For 2.1 or later.

Let's revisit this next week.

[freedomtan]

@mohitmundhragithub: Yes @Mostelk: by default "NPU", changed it when necessary @d-uzlov tflite backend(s) from settings.h Intel: undecided. Google: fine.

Generally, yes. @anhappdev please do it.

[anhappdev]

To let the backend report the accelerator name back to the app, I suggest we add a new API named mlperf_backend_accelerator_name, similar to the existed API mlperf_backend_vendor_name. In the API, the backend vendor can use a static value or variable as wished.

This change requires modifying code in the module android/cpp. So I am waiting to finish the issue https://github.com/mlcommons/mobile_app_open/issues/249 first.