Add delegate options to the TFLite backend

[freedomtan]

Currently, we check if the NNAPI Delegate works first. If yes, we use it. If no, falling back to CPU (TFLite interpreter). It would be more convenient if uses could check if she/he wants to use

1. NNAPI delegate,
2. GPU delegate,
3. TFLite interpreter (CPU), or
4. any other 3rd-party delegates

[freedomtan]

1. current statua: q , s, p, n, and TFLite
2. To be: 2.1 each backend: if the backend wants to support default TFLite fallback. Q. SNPE, 2.2. TFLite backend <--- add options,

@freedomtan add most descriptions for what was proposed for vendor backend and the "fallback" tflite backend.

[freedomtan]

Current status: platforms / devices supported by current app is quit limited. Our vendors backends support most recently high-end or middle-end devices only. And vendor backends probably do support new high-end devices with high-end SoCs (because SoC vendors have not submitted newest patches / settings yet. To mitigate this problem, we suggest to add

1. TFLite interpreter fallback to each vendor backend, so that each vendor backend could support more platforms, and
2. More options to the "fallback TFLite backend".

To be: Suppose we have a phone with Samsung Exynos 9999,

1. Qualcomm backend will return false when mlperf_backend_matches_hardware() is called
2. Samsung backend returns false now. We probably want change it to return true and add a warning or alter message saying "hey, it's a Samsung platform, but we are gonna run on CPU with TFLite interpreter for now". The user could select to continue or not.
3. If yes, vendor backend handle it. If no, continue to the "fallback TFLite backend".

If there is a new from Oppo with Oppo's SDK, we go to the "fallback TFLite backend".

If there is a MediaTek SoC, but the MediaTek backend just refuses to support it. Then we go to the "fallback TFLite backend", too.

[anhappdev]

I would propose an alternative flow to choose the backend: Let say an Android phone with supported Samsung hardware.

1. The app starts and checks which backends are available (mobile_back_samsung and mobile_back_tflite are available)
2. The apps can either start with the mobile_back_samsung or ask the user which backend he wants to use.
3. The user can change the selected backend in the settings screen.

---

Regarding choosing a delegate after choosing a backend: I know that TFLite has multiple delegates we can choose from, but what about other backends? We will probably need an API to ask the backend if it has multiple delegates and an API to switch to that selected delegate.

[freedomtan]

a case: suppose we have a new Exynos 9000 SoC: the choices to run TFLite or any other fallback stuff should be decided or say controlled by the Samsung backend.

[freedomtan]

@freedomtan to check if we can ask a Google TFLite guy to talk to us (about delegate related stuff)

[anhappdev]

@Mostelk @freedomtan @AhmedTElthakeb @mohitmundhragithub The flow we discussed in the meeting today:

1. The app starts and checks which backends are available.
2. vendor_backend can decide either: 2.1 the phone is supported, continue with vendor_backend 2.2 the phone is not supported, show an error to user that this phone is not supported 2.3 the phone is not supported, continue with fallback mobile_back_tflite

[freedomtan]

@anhappdev: make the choice of TFLite delegates (NNAPI delegate, GPU delegate, and CPU only) available to users.

2.2 is not preferred. 2.2.5: tflite interpreter + whatever options you want to add in the vendor backend as the latest line of defense, then you don't need to go to 2.3.

outline/define the interface for users to choose as @mohitmundhragithub mentioned. Hopefully, we can use the same component for both vendor backend and the mobile_back_tflite.