The ML hardware accelerator is still developing quickly and it is hard to group them all into one single category (not like GPU which has become comparatively stable). Should we expose some interface to let the client control whether and what such SoC will be used for inference?
The pros is that a ML model may or may not be suitable for a particular SoC so a user can explicitly control this for optimal performance. But the cons are big:
This may introduce strong privacy issue (i.e. the client needs to know which SoCs are available?)
This is rather complex, not sure whether web developers can correctly handle this.
The API will be much more complex.
So I personally prefer not to expose such interface. Instead, besides "CPU" and "GPU", maybe we can provide a third type of device preference "Auto", for which, we will use some heuristic to transparently apply the ML SoCs when it is available and appropriate.
The ML hardware accelerator is still developing quickly and it is hard to group them all into one single category (not like GPU which has become comparatively stable). Should we expose some interface to let the client control whether and what such SoC will be used for inference?
The pros is that a ML model may or may not be suitable for a particular SoC so a user can explicitly control this for optimal performance. But the cons are big:
So I personally prefer not to expose such interface. Instead, besides "CPU" and "GPU", maybe we can provide a third type of device preference "Auto", for which, we will use some heuristic to transparently apply the ML SoCs when it is available and appropriate.